Proposal for new branching strategy for keripy, KERIA and SigPy

[pfeairheller]

This is a proposal to update the branching, tagging and releasing strategy for KERIpy, KERIA and SigPy. Other repos, specifically SigTS are welcome to adopt this new strategy as well but it is not mandatory.

Motivation

We originally selected GitFlow as our branching strategy for KERIpy as we added new developers and started releasing software that was being used in other repos (KERIA, SigPy) and in production. This choice was driven mainly by what the maintainers were familiar with. Unfortunately, that experience was with closed source projects without uncontrolled third party dependencies on our work. The current strategy is unwieldy, is causing more work than necessary and, in one instance, a lost feature. To make matters worse, there are circumstances we can't currently handle without a change / clarification of approach.

GitFlow

Our current model attempts to follow the GitFlow workflow which relies on two long living branches, main and development. All feature work branches off of development and when complete, pull requests are open against development. Once we are ready for a new "release" we create a release branch from development that is tested for some period of time before being merged into main and the back into development. Release tags and artifacts (PyPi package and docker images) are created at the HEAD of main. In the event that we need to create a hotfix for the current "release", we create a branch off of main, apply the fix and create a PR to merge it back into main. Release tags and artifacts are once again created at the HEAD of main. At this point, the changes can be merged back into development if appropriate.

Challenges with GitFlow

Due to the rapid pace of development both on current feature work as well as patches to the current 1.1.* release, we have lots of divergent code being changes in both the development and main branches. When we need to cut a new release branch from development that will get merged into main, the merge results in many hard to resolve conflicts. As mentioned above, we have already lost one feature as a result of resolving these conflicts. In addition, merging in the the release branches has become very time consuming for the core maintainers.

Another unresolved problem we do not currently have a solution for is when a patch is required for older versions of released software. For example, we currently have a 1.0.0 release that is a tag in main as some point in the past. We subsequently released 1.1.0 which was also tagged on main. As we've added patches to 1.1.*, we've created hotfix branches off of main and when merged back in, we tag and created release artifacts from main (we then cherry pick those changes back into development when appropriate.). What we have not defined is how to create a patch for 1.0.0. The main branch has moved on so the only solution would be to create a long living branch from the 1.0.0 tag, apply the fix, tag as 1.0.1 and create the release artifacts from this new long living branch. From this point forward, we have a home for future patches in the 1.0.* release "fork" (sorry for using the overloaded term).

This approach to patching old versions seems like the correct approach (and one we propose to use below) but it gets murky when describing the approach relative to GitFlow. We will end up using a modified version of GitFlow to solve this problem and still have the other complications described above.

Feature Based Workflow

Our proposed solution is to switch to what I've seen referred to as a "Feature Based Workflow". Day to day work is nearly identical to our current approach with one main (pun intended) exception. We will only have one long living branch for "normal" development. We will remove the development branch and all feature branches will be created off of and PR'ed / merged back into the main branch. At all times, the code on the main branch must be "production ready". The key difference from our current GitFlow approach is that main will no longer be synonymous with the current release of the software.

At any point in time, we can decide to create a new release from main by tagging and creating the release artifacts without having to go through a lengthy release branch, merge processes.

With this new approach in place we can apply the patch approach mentioned above for creating new point releases of the software. The result will be long living branches for any release of the software that need patch releases. The advantage here is that developers who create a new patch release can make the decision at the time of patching where else the code needs to be applied, when the change is fresh in mind. As opposed to the current approach where the patch gets cherry picked back into the development branch and then, at some time later merged into main where conflicts may arise long after the change was made.

Normal, Daily Feature Workflow

Below are the steps for creating a new feature intended for "the next new release of the software". These steps assume you are working from a forked repo that you have write access to. I will leave out the specific details of working with your fork as that is tangential to our new workflow.

1. Pull the latest main branch.
2. Create a new branch from main for your new feature.
3. Write amazing code for your new killer KERIpy feature.
4. Ensure all existing unit tests and scripts still pass and add test coverage for your new feature.
5. Commit your changes, push your new branch and create a PR against main.
6. Wait for the wonderful maintainers to approve and merge your branch into main.
7. Fetch and merge new version of main locally, delete your feature branch, rather, rinse, repeat.

Step 6 becomes a feedback loop if changes are requests from the wonderful maintainers. You would make the suggested changes, re-push your branch and wait for more feedback or for your PR to be merged.

New Release Workflow

The steps listed in this section are to be used when creating a new release (*.0.0 or *.*.0) from the current main HEAD. The commands in this section assume you have named the git remote for WebOfTrust/keripy upstream.

1. Pull the latest main branch and ensure all tests and test scripts pass.
2. Update the version information in setup.py, Makefile and src/keri/__init__.py.
3. Commit your changes with git commit -a -m "New version comment".
4. Create the new tag for the release with git tag -a *.*.0 ensuring that the new tag is greater than (in semantic versioning sense) any existing tag.
5. Push the new tag with git push --tags upstream main.
6. Create and publish new PyPi package using the directions in setup.py.
7. Create and publish new Docker image using the Makefile
8. Create a new unbounded term branch for future patches name with an r the first two numbers in the version and an x as the last character using checkout -b <branch_name> tags/<tag_name>. So for the 2.3.0 release, the new branch name will be r2.3.x and the git command will be checkout -b r2.3.x tags/2.3.0.
9. Push the new branch to upstream

Patch Workflow

These steps are for creating a new patch version of the form *.*.(*>0). This requires the developer to checkout the unbounded term branch for all patch releases of *.*.x. The commands in this section assume you have named the git remote for WebOfTrust/keripy upstream.

1. Checkout the r*.*.x branch. So if you were adding a patch to the 2.3.0 version, the command git checkout r2.3.x will be used.
2. Apply your super special code changes, commit and push.
3. Create a PR against the r*.*.x branch and wait for the awesome maintainers to merge your PR.
4. Pull the latest version of the r*.*.x branch after your PR has been merged
5. Update the version information in setup.py, Makefile and src/keri/__init__.py.
6. Commit your changes with git commit -a -m "New version comment".
7. Create the new tag for the release. For example, if you are creating the first patch for 2.3.0 it use git tag -a 2.3.1.
8. Push the new tag with git push --tags upstream rx.x.1.
9. Create and publish new PyPi package using the directions in setup.py.
10. Create and publish new Docker image using the Makefile

Downstream Effects

Ultimately these changes will result in the following that will effect those that depend on these repositories:

1. There will no longer be an unbounded term branch named development
2. To work with the code for a specific version, checkout the r*.*.x branch.
3. main HEAD should no longer be considered synonymous with the latest released version of the code. Code on main will always be production ready, but may not represent a release of the software.

[lenkan]

This is a very good proposal in my opinion. I would fully support adopting this strategy in signify-ts as well.

[rodolfomiranda]

+1

[2byrds]

Thank you for the detailed discussion. The approach seems reasonable and I'm glad to see strong support.

[henkvancann]

A new strategy could use some exam to test whether you're fit to get involved?

https://hackmd.io/VFtLabBtQbypI3Mhak1iuQ

This is chatGPT generated and I wonder if it's useful. Please comment with your 5 favorite questions and 5 silly questions out of this collection.

[m00sey]

I think this makes sense.

[kentbull]

This sounds like trunk-based development with long-lived release branches. It makes sense to me. Maintaining release branches seems inescapable given that different products build against different versions of KERIpy. Keeping long-lived release branches around for patches seems required, yet it will require discipline to keep things in sync with main.

[tibor19]

Why not use "folders" for releases? So instead of having all the release branches called r*.*.x we can name them release/*.*.x. This way we can make it easier to filter the branches. We can even extend that and use feature/, bug/ and so on.

[VasiliyS]

@pfeairheller

main HEAD should no longer be considered synonymous with the latest released version of the code. Code on main will always be production ready, but may not represent a release of the software.

What about breaking changes, such as the one we are discussing in the #842?

[pfeairheller]

They should be fixed before the next release of the software. No one should be pointing to main for anything but testing compatibility and reporting breaking changes.