PEARL XIV : An Inquiry on Release/Hardening sprint in Scrum
There is a deep divide between people who recognize that spending some time on hardening is needed for many environments, and people who are adamant that allocating some time for hardening is a sign that you are doing some things – or everything – wrong
A Hardening/release sprint, often part of a project using agile management methodologies, incorporates activities, sometimes not related to creating product features, that the development team can’t realistically complete within development sprints. To accommodate prerelease activities and help ensure that the release goes well, scrum teams often schedule a release sprint as the final sprint prior to releasing product to customers.
There are two components of the Production Release practice: 1) Release Preparation, and 2) Deployment. Release preparation establishes a release baseline and produces all the necessary supporting material necessary to deploy (and back out, if necessary) the release.
Deployment involves the act of delivering the release into the production environment, verifying that the integration of the release package into the existing environment was successful, and notifying all relevant stakeholders that the features of the release are available for use.
The Hardening release sprint should contain anything you need to do to move the working product to production. Sprint backlog items in a release sprint may include
- Creating user documentation for the most recent version of the product
- Performance testing, load testing, security testing, and any other checks to ensure the working software — or other product — will perform acceptably in production
- Integrating the product with enterprise-wide systems, where testing may take days or weeks
- Completing organizational or regulatory procedures that are mandatory prior to release
- Preparing release notes — final notes about changes to the product
If your product is software, backlog items for the Hardening release sprint may also include
- Preparing the deployment package, enabling all the code for the product features to move to production at one time
- Deploying your code to the production environment
In a hardening sprint, the team stops focusing on delivering new features or architecture, and instead spends their time on stabilizing the system and getting it ready to be released.
For some people, hardening sprints are for completing testing and fixing work that couldn’t be done – or didn’t get done – earlier. This might include UAT or other final acceptance testing if this is built into a contract or governance model.
Mike Cohn recognizes that teams may need a “ hardening/release sprint” at the end of each release cycle, because the team’s definition of “done” may not be enough – that a “potentially shippable product”and a system that is actually “shippable” or ready for production aren’t the same thing. He suggests that after every 3-5 feature iterations, the team may want to schedule a release sprint to do work like expensive manual system and integration testing and extra reviews, whatever is needed to make sure that what they think is done, is actually done.
Anand Viswanath, in “The end of regression, stabilisation, hardening or release sprints”, describes a common approach where teams schedule 1 or 2 stabilization sprints every 4-6 iterations to do regression testing and system testing in a staging environment, and then fix whatever bugs are found. As he points out, it’s hard to predict how much testing might be required and long it will take to fix whatever problems are found, so the idea is to time box this work and then triage the results.
Because this can be an expensive and risky and stressful way to work, Vishwanath recommends following Continuous Delivery to build an automated test pipeline through to staging in order to catch as many problems as early as possible. This is a good idea, but most large projects, especially projects starting from a legacy code base, will still probably need some kind of hardening or integration testing phase at regular points regardless of what kind of continuous testing they are doing.
Some testing, like interoperability testing with other systems and operational testing, can’t be done effectively until later, when there is enough of a working system to do end-to-end testing, and some of this testing can only be done in staging , or in production. For some systems, load testing and stress testing and soak testing also needs to be left to later, because these teams don’t have access to a right high end test system to run high load scenarios before they get to production.
Is Hardening a sign that you aren’t doing things right?
Not everyone thinks that scheduling a hardening sprint for testing and fixing like this is a good idea:
“[a hardening sprint] might take the cake for stupid things invented that has lead to institutionalized delusion and ‘Agile’ dysfunction.” Janelle Klein, Who Came up with the “Hardening Sprint”?
For many people, a hardening sprint or release sprint is a bad “process smell”: a sign that the team isn’t working properly or thinking clearly:
“The problem with “hardening sprints” is that you are lying. You make believe your imaginary burndown during the initial sprints shows that you are approaching Done. But it’s a lie–you aren’t getting any closer to being ready for Production until you begin your Test phase. You wrote a pile of code that you didn’t test adequately. You don’t know how good it is, you don’t know how much work you have left to do, and you don’t know how much longer it will take, until you are deep into your Test phase.” Richard Kasperowski, Hardening sprints? Sorry, you’re not Agile
Ron Jeffries says that a hardening sprint for testing and fixing is a clear anti-pattern. Many Agilists agree: if you need a separate sprint to fix bugs, then you’re doing something wrong. But that doesn’t mean that you won’t need extra time to fix things before the system goes live – knowing that it is wrong doesn’t make the bugs go away, you still have to fix them. As the same discussion thread points out, there is a risk that your “definition of done” could fall short of what is actually needed by the customer, so you should plan for 1 or more hardening sprints before release, to double-check and stabilize things, just in case.
In these cases, the need for hardening sprints is a sign of a team’s immaturity (from a post by Paul Beavers):
- A beginning agile team will prefer to schedule 6 hardening iterations after a 12 iteration development plan. This is “agile” to the hard core “waterfall guy”.
- As time goes by, the team will mature a bit and you will see the seasoned agile team will shrink the number of required hardening iterations at the end, just because they understand they need to “fix” the high severity bugs as they go and QA understands they need to test closer and better early up in the release cycle.
- Further down the road the team will notice that by adding a hardening iteration in the middle of the development cycle (and flushing out even lesser priority bugs earlier on in the process), it will help them to maintain cadence later on.
- The final step of maturity is there when the team starts understanding “hardening is not required any more”, because they made fixing bugs part of their daily routines.
Hardening is whatever you need to do to Make the System Ready for Production
Another way of looking at hardening, is that this is when you stop thinking about features and focus all of your time on the detailed steps of deploying, installing and configuring the system and making sure that everything is working from end-to-end. In a hardening sprint, your most important customers are operations and support, the people who are going to make sure that the system is running, rather than the end users.
For some teams, this kind of hardening can come as an ugly and expensive surprise, after they understand that what they need to do is to take a working functional prototype and make it ready for the real world:
“All those things that got skipped in the first phase – error handling, monitoring, administration – need to get put into the product.” Catherine Powell, The “Hardening Myth”
But a hardening sprint can also be when when you take care of what operations calls hardening: reviewing and preparing the production environment and securing the run-time, tightening up access to production data, double-checking system and application configs, making sure that auditing is enabled properly, wiring the system in to operations monitoring and metrics collection, checking system dependencies like platform software versions and patch levels (and making sure that all of the systems are consistent, that there aren’t any snowflakes), completing final security reviews and other review and release gates, and making sure that the people installing and running the software have the correct instructions.This is also when you need to prepare your roll-back plan or recovery plan if something bad happens with the release, and test your roll-back and recovery steps. Walk through and rehearse the release process and checklists, and make sure that everyone is prepared to roll out patches quickly after the release is done.
Jail Free Card
Agilists have also heard of hardening sprints being used as a sort of “get out of jail free card” within Scrum teams. The conversations are usually along the following lines:
We don’t have enough time to test that component properly, so we’ll do it in hardening
We don’t have time to fix all of the cosmetic bugs we produced in this sprint, so we’ll do it in hardening
Let’s defer customer support training entirely to hardening
The design is still too volatile to document. We’ll wait until it stabilizes and document it in hardening
What is happening here has to be noted ? Hardening can undermine the broad definition of done that is so incredibly important to Scrum’s quality and delivery dynamics. But if it is so bad, why do it at all? That is why there are some of the contexts where hardening might not only be a good idea, but almost required.
Hardening is something that you have to do
Some people see an obvious need for hardening sprints. For example, Dean Leffingwell includes hardening sprints in his “Scaled Agile Framework”, because there is some work that can only really be done in a final hardening phase:
- Final exploratory and field testing
- Checklist validation against release, QA and standards governance
- Release signoffs if you need them
- Ops documentation
- Deployment package
- Communicate release to everyone (hard to do in big companies)
- Traceability etc for high assurance and regulatory compliance
Leffingwell makes it clear that hardening shouldn’t include system integration, fixing high priority bugs, automating test scripts, user documentation, regression testing and code cleanup. There is other work that should be done earlier – but in the first year or so, will probably need to be done in a late hardening phase:
- Cross-component integration, integration with third-party/customer
- Integrated system-level testing
- Final QA sign-offs
- User doc finalization
Dan Rawsthorne explains that teams need at least one release sprint at first to get ready for release to production, because until you’ve actually done it, you don’t really know what you need to do. Release sprints include tasks like:
- Exploratory testing to double check that key features are working properly
- Stress testing/load testing/performance testing – testing that is expensive to setup and do
- Interoperability testing with other production systems
- Fix whatever comes out of this testing
- Review and finish off any documentation
- Train support and sales and customers on new features
- Help with press releases and other marketing material
The Software Project Manager’s Bridge to Agility anticipates that teams will need at least a short hardening iteration before the system is ready for release, even if they frontload as much testing as possible. A release iteration is not a test-fix phase – it’s when you prepare for the release: capturing screenshots for marketing materials, final tests, small tweaks, finish documentation for whoever needs it, training. The authors suggest however that if some developers have any time left over in the release iteration, they can do some refactoring and other cleanup – which Agilists think is bad advice, given that at this point you don’t want to be introducing any new variables or risks.
Disciplined Agile Delivery, a method that was developed by Scott Ambler at IBM to scale Agile practices to large organizations and large projects, includes a Transition Phase before each release to take care of:
- Transition planning and coordination
- End-of-lifecycle testing and fixing
- Testing and rehearsing deployment
- Data setup and migration
- Pilots and beta testing (short UAT if necessary)
- Reviewing and finalizing documentation
- Preparing operations and support
- Stakeholder training
This kind of transition can take almost no time, or it can take several weeks, depending on the situation.
Hardening – taking some time to make sure that the system is really ready to be released – can’t be avoided. The longer your release cycles, the further away development is from day-to-day production, the more hardening you need. Even if you’ve been doing disciplined testing and reviews in stream, you’re going to find some problems at the end. Even if you planned ahead for transition, you’re going to run into operational details that you didn’t know about or didn’t understand until the end.
When agilists first launched a platform from start up, they had to do hardening and stabilization work before going live to get the system ready, and some more work afterwards to deal with operational issues and requirements that they weren’t prepared for. They included time at the end of subsequent releases for extra testing, deployment and roll back planning, and release coordination.
But as they shortened their release cycle, releasing less but more often, and as they built more fail-safes into the system and as they learned more about what they needed to do in ops, and as they invested more in simplifying and automating deployment and everything else that they could, they found that they didn’t need time any outside of regular iterations for hardening. They a’re still doing hardening – but now this is part of the day-to-day job of building and releasing software.
But there are a whole lot of contexts where hardening can be incredibly useful. Here are a few of these, just to set you up for healthier applications:
1. Distributed and At-Scale Agile: It is quite easy to say that a team should “integrate” their software fully within each sprint. But what if you have 20-50 teams working on the same project and the teams are geographically distributed?
Sure, you want to try to integrate as much as possible across the teams, but clearly you are not going to do it all. It just does not meet reasonable ROI on a sprint-by-sprint basis.
In these cases, having a hardening sprint that is focused toward full Integration, regression, and system testing might be a prudent trade-off.
.2. Customer Receptivity: Something that gets lost in the “potentially shippable product increment” goal that is such a strong part of Scrum is – what if your customer simply cannot tolerate or accept a release every 1-2-3 weeks? What do you do then? Many domains and customers simply cannot.
The game is then to accumulate partial release contents over the course of several-to-many sprints. But then there is the need to re-qualify all of that code when you are ready
to do a real (not potential) release.
3. Test Automation Coverage: Having high degrees of test automation, and automation in general, is one of the ways that agile teams can truly go fast. It provides a wonderful development safety net and gets feedback to the team quickly. But what if you do not have it? What if you are currently stuck with thousands of manual, albeit valuable, test cases to run? Clearly if you run all of them within each sprint you would derail the focus. But running risk-based regression and functional testing is risky since you are deferring levels of coverage. A hardening sprint is a solid way to handle this technical test debt before releasing.
4. Skewed Sprint Consolidation: In many agile contexts, some work is done a “sprint ahead” of other activities. For example, UX design is often completed by a design Scrum
team and then “handed off” to a front-end implementation team for coding in a follow-on sprint. This sort of staging often happens with research (story spikes), design, architecture, and other up-front activities in projects. The hardening sprint is a place to check that all of these skewed activities have come together into a cohesive package from a customer perspective. It certainly should not be the first or only time, but it can serve as a design convergence point.
5. Defect Rework: At the end of the day, bugs are discovered pre-release whether you are implementing Waterfall or Agile methods. If you are doing Agile well, then you are finding fewer and the rework cycle time is much shorter. But, inevitably, some cleanup of bugs pre-release is fairly normal. Reserving time in your hardening sprint gives you some time for final defect cleanup before making the release.
6. Deployment Readiness and Training: There is a “whole lot” of work required to get a release out the door in many organizational contexts and business domains. Agilists think one of the things that “potentially shippable product increment” does is to trivialize this level of pre-release preparation. While thier definition of hardening sprints tends to be more testingand defect-repair focused, it is an interval that allows time to prepare software for, err, release! Dan Rawsthorne has written an article on the Agile Atlas site entitled Out the
Door: the Release Sprint, which is perhaps another name for a hardening sprint. Dan focuses the sprint on this area though –finishing all of the details surrounding actually
releasing a supportable product.
Regulations, Governance, and the Art of Trivialized Agile Testing: Finally, another focus for a hardening sprint is making all of the steps and creating all of the obliged artifacts that meet your internal and external regulations for the product. Consider it a “governance acceptance test” checklist where various requirements are exercised and confirmed. Quite often this includes a full regression test that provides traceability between the project requirements (User Stories) and the tests that covered that functionality.
In many domains, financial and healthcare for instance, proof of completeness and following your predefined processes are required steps pre-release
These sorts of sprints are considered a Scrum-butt or Scrum anti-pattern by many of the leading Scrum authorities (CST’sand CSC’s)