Crispin, Lisa - Agile Testing - A Practical Guide for Testers and Agile Teams

Agile development doesn’t work if stories, iterations, or releases aren’t “done.” “Doneness” includes testing, and testing is often the thing that gets postponed when time is tight. Make sure your success criteria at every level includes all of the necessary testing to guide and validate development.

Each project, each team, each business is unique. Agile teams work with the business experts to decide when they’re ready to deliver software to production. If the release deadline is set in stone, the business will have to modify scope. If there’s enough flexibility to release when the software has enough value, the teams can decide when the quality criteria have been met and the software can go to production.

Challenging Release Candidate Builds

Coni Tartaglia’s team uses a checklist to evaluate each release candidate build. The checklist might specify that the release candidate build:

• Includes all features that provide business value for the release, including artwork, logos, legal agreements, and documentation

• Meets all build acceptance criteria

• Has proof that all agreed-upon tests (acceptance, integration, regression, nonfunctional, UAT) have passed

• Has no open defect reports

Coni’s team challenges the software they might ship with a final set of inspections and agreed-upon “release acceptance tests,” or “RATS.” She explains:

The key phrase is “agreed-upon tests.” By agreeing on the tests in advance, the scope for the release checklist is well defined. Include system-level, end-to-end tests in the RATS, and select from the compatibility roster tests, which will really challenge the release candidate build. Performance tests can also be included in RATs. Agree in advance on the content of the automation suites as well as a subset of manual tests for each RAT.

Agree in advance which tests will be repeated if a RAT succeeds in causing the failure of a release candidate build. If the software has survived several iterations of continuously run automated regression tests, passing these final challenges should be a breeze.

Defining acceptance criteria is ultimately up to the customers. Testers are in a unique position to help the customer and development teams agree on the criteria that optimize product quality.

Traditional software development works in long time frames, with deadlines set far in advance and hurdles to clear from one phase to the next. Agile development lets us produce quality software in small increments and release as necessary. The development and customer teams can work closely to define and decide what to release and when. Testers can play a critical role in this goal-setting process.

Release Management

Many organizations have a release management team, but if you don’t, someone still does the work. Many times in a small organization it is the QA manager who fulfills this role. The person leading the release may hold a release readiness meeting with the stakeholders to evaluate readiness.

A release readiness checklist is a great tool to use to walk through what is important to your team. The intention of this checklist is to help the team objectively determine what was completed and identify the risks associated with not completing a task.

For example, if training is not required because the changes made to the product were transparent to the end user, then the risk is low. However, if there were significant changes to the process for how a new user is created in the system, the risk would be very high to the production support or help teams, and may warrant a delay. The needs of all stakeholders must be considered.

Release notes are important for any product release. The formality of these depends on the audience. If your product is aimed at developers, then a “read me” text file is probably fine. In other cases, you may want to make them more formal. Whatever the media, they should address the needs of the audience. Don’t provide a lot of added information that isn’t needed.

When Janet gets a new release, one of the first things she does is check the version and all of the components. “Did I get what they said they gave me? Are there special instructions I need to consider before installing, such as dependencies or upgrade scripts?” Those are good simple questions to answer in release notes. Other things to include are the new features that the customer should look for.

Release notes should give special consideration to components that aren’t part of what your development team delivered, such as a help file or user manuals prepared by a different team. Sometimes old release notes get left on the release media, which may or may not be useful to the end user. Consider what is right for your team and your application.

Packaging

We’ve talked a lot about continual integration. We tend to take it for granted and forget what good configuration management means. “Build once, deploy multiple times” is part of what gives us confidence when we release. We know that the build we tested in staging is the same build that the customer tested in UAT and is the build we release to production. This is critical for a successful release.

If the product is intended for an external customer, the installation should be easy, because the installation may be the first look at the product that customer has. Know your audience and its tolerance level for errors. How will the product be delivered? For example, if it is to be downloaded off the Internet, then it should be a simple download and install. If it is a huge enterprise system, then maybe your organization needs to send a support person with the product to help with the install.

Customer Expectations

Before we spring new software on our customers, we’d better be certain they are ready for it. We must be sure they know what new functionality to expect and that they have some means to deal with problems that arise.

Production Support

Many organizations have a production or operations support team that maintains the code and supports customers after it’s in production. If your company has a production support team, that group is your first customer. Make it your partner as well. Production support teams receive defect reports and enhancement requests from the customers, and they can work with your team to identify high-risk areas.

Very often the production support team is the team that accepts the release from the development team. If your organization has this type of hand-off, it is important that your development team works closely with the production support team to make it a smooth transition. Make sure the production support team understands how to use the system’s log files and the messaging and monitoring systems in order to keep track of operations and identify problems quickly.

Understand Impact to Business

Every time a deployment to production requires an outage, the product is unavailable to your customer. If your product is a website, this may be a huge impact. If your product is an independent product to be downloaded onto a PC, the impact is low. Agile teams release frequently to maximize value to the business, and small releases have a lower risk of a large negative impact. It’s common sense to work with the business to time releases for time periods that minimize disruption. Automate and streamline deployment processes as much as possible to keep downtime windows small. A quick deployment process is also helpful during development in short iterations where we may deploy a dozen times in one day.

International Considerations

Markus Gärtner, an “agile affected” testing group lead, explains his team’s approach to timing its releases: