Thursday, October 1, 2009

Large Team, 6 week Release Cycle...... Transition between Releases

In the past 5 years that I have been working with Agile teams (10-25 members; 4-7 month release cycle), I never had to think too much about transitioning the team from release n to release n+1. The transition has been quite simple so far. A release typically consists of a number of iterations followed by a regression and/or UAT phase. During the regression period the team will be fixing defects and stabilizing the application for deployment to production. As we progress into the regression and/or UAT phase, the number of defects reduce. At that point of time, we could branch off for the release. A small part of the team would continue to fix defects on the release branch. The number of defects fixed after branching is not too big. With regular merges from branch to trunk, we could easily make sure that all changes from branch are available in trunk as well.

The current team that I am working with, is 190 people strong. We have been working on this code base for more than 2 years. We follow a 6-8 week release cycle i.e. a release is deployed to production every 6 - 8 weeks. The above mentioned model/process fails because of the amount of changes that that this team accomplishes per day. Since the team and code base are big, we may fix up to 30 defects per day. If I try to replicate the process/model mentioned above for a small/mid size team, the merge becomes very painful as the amount of changes on trunk as well as the branch are too big to be merged effectively. Most of the merges will result in some defects either getting reopened or some new defects getting introduced. In order to reduce this problem, we should branch as late as possible. At the same time, if we have the entire team fixing defects, it is impossible to find work for all the dev-pairs on the team. So, from this perspective, we should branch as early as possible.

In order to make the transition as smooth as possible, we needed to get a period where we could fix defects and stabiles the build and at the same time continue to develop new functionality without impacting the release. Rajiv and I started looking into our story backlog to check if we can find such stories. We soon found that such stories do exist. Any story that is impacting/changing a feature not going live in the current release falls under this category. In our case, the application has 15 portals. If a portal is not getting delivered as part of release n, then we can continue to make changes without impacting the stability of the application. Other easy example is if a feature is hidden behind a permission and will not be delivered as part of release n, we can play stories that change/evolve this feature. While identifying these stories we need to evaluate if the story has a potential to impact other features as well. If the answer is yes, then such story should not be played before branching.

This approach provides and optimal balance between the need to branch as early and as late as possible. Using this approach we can ensure a smooth flow of stories through the pipeline even when the team transitions from release n to release n+1. The team never starts from an empty pipeline and thus the entire system is geared to maximize the throughput.

In addition to the above mentioned approach, GIT has helped us to considerably reduce the effort and time required for merging from a branch to trunk or across branches. I will share our experience of working with GIT in my next post.