Crispin, Lisa - Agile Testing - A Practical Guide for Testers and Agile Teams

Deliverables are not always for the end customer, and they aren’t always in the form of software. There are many internal customers, such as the production support team members. What will they need to make their job easier? Workflow diagrams can help them understand new features. They would probably like to know if there are work-arounds in place so they can help customers through problems.

Janet often gets asked about test coverage of code, usually by management. How much of the application is being tested by the unit tests or regression tests? The problem is that the number by itself is just a number, and there are so many reasons why it might be high or low. Also, code coverage doesn’t tell you about features that might have been missed, for which no code exists yet. The audience for a deliverable such as code coverage should not be management, but the team itself. It can be used to see what areas of the code are not being tested.

Training could be considered a deliverable as well. Many applications require customized training sessions for customers. Others may only need online help or a user manual. Training could determine the success of your product, so it’s important to consider. Lisa’s team often writes task cards for either a tester or the product owner to make sure training materials and sessions are arranged. Some people may feel training isn’t the job of testers or anyone else on the development team. However, agile teams aim to work as closely as possible with the business. Testers often have the domain expertise to be able to at least identify training that might be needed for new or updated features. Even if training isn’t the tester’s responsibility, she can raise the issue if the business isn’t planning training sessions.

Many agile teams have technical writers as part of the team that write online help or electronic forms of documentation. One application even included training videos to help get started, and different members of the team were the trainers. It is the responsibility of the team to create a successful product.

Nonsoftware Deliverables

Coni Tartaglia, software test manager at Primavera Systems, Inc., reflects on what has worked for her team in delivering items that aren’t code but are necessary for a successful release.

Aside from the software, what is the team delivering? It is helpful to have a conversation with the people outside of the development team who may be concerned with this question. Groups such as Legal, Product Marketing, Training, and Customer Support will want to contribute to the list of deliverables.

After there is agreement on what is being delivered, assembly of the components can begin, and the Release Management function can provide confirmation of the delivery through execution of a release checklist. If the release is an update to an existing product, testers can check the deliverables from previous releases to ensure nothing critical is left out of the update package. Deliverables can include legal notices, documentation, translations, and third-party software that are provided as a courtesy to the customers.

Agile teams are delivering value, not just software. We work together with the customer team to improve all aspects of the product.

There are no hard and fast rules to what should be delivered with the product. Think of deliverables as something that adds value to your product. Who should be the recipient of the deliverable, and when does it make the most sense to deliver it?

Releasing the Product

When we talk about releasing the product, we mean making it available to the customer in whatever format that may take. Your organization might have a website that gets updated or a custom application that is delivered to a few large customers. Maybe the product is shrink-wrapped and delivered to millions of PCs around the world, or downloaded off the Internet.

Release Acceptance Criteria

How do you know when you’re done? Acceptance criteria are a traditional way of defining when to accept the product. Performance criteria may have to be met. We capture these for each story at the start of each iteration, and we may also specify them for larger feature sets when we begin a theme or epic. Customers may set quality criteria such as a certain percentage of code covered by automated tests, or that certain tests must pass. Line items such as having zero critical bugs, or zero bugs with serious impact to the system, are often part of the release criteria. The customers need to decide how they’ll know when there’s enough value in the product. Testers can help them define release criteria that accomplish their goals.

Agile teams work to attain the spirit of the quality goals, not just the letter. They don’t downgrade the severity of bugs to medium so they can say they achieved the criterion of no high-severity bugs. Instead, they frequently look at bug trends and think of ways to ensure that high-severity bugs don’t occur in production.

Your quality level should be negotiated with your customer up front so that there are no unpleasant surprises. The acceptance tests your team and your customers defined, using real examples, should serve as milestones for progress toward release. If your customer has a very low tolerance for bugs, and 100% of those acceptance tests must be passing, your iteration velocity should take that into consideration. If new features are more important than bug fixes, well, maybe you will be shipping with bugs.

A Tale of Multitiered “Doneness”

Bob Galen, agile coach and author of Software Endgames , explains how his teams define release acceptance criteria and evaluate whether they’ve been met.

I’ve joined several new agile teams over the past few years, and I’ve seen a common pattern within those teams. My current team does a wonderful job of establishing criteria at a user story or feature level—basically defining acceptance criteria. We’ve worked hard at refining our acceptance criteria. Initially they were developed from the Product Owners’ perspective, and often they were quite ambiguous and ill-defined. The testers decided they could really assist the customers in refining their tests to be much more relevant, clear, and testable. That collaboration proved to be a significant win at the story level, and the Product Owners really valued the engagement and help.

Quite often the testers would also automate the user story acceptance tests, running them during each sprint but also demonstrating overall acceptance during the sprint review.

One problem we had, though, was getting this same level of clarity for “doneness” at a story level to extend beyond the individual stories. We found that often, when we approached the end of a Sprint or the end game of a release, we would have open expectations of what the team was supposed to accomplish within the sprint. For example, we would deliver stories that were thoroughly tested “in the small”; that is, the functionality of those stories was tested but the stories were not integrated into our staging environment for broader testing. That wasn’t part of our “understanding,” but external stakeholders had that expectation of the teams’ deliverables.

The way the teams solved this problem was to look at our criteria as a multitiered set of guiding goals that wrap each phase, if you will, of agile development. An example of this is shown in Table 20-1.

Table 20-1 Different Levels of Doneness

Defining doneness at these individual levels has proven to work for our teams and has significantly improved our ability to quantify and meet all of our various customer expectations. Keep in mind that there is a connection among all of the criteria, so defining at one level really helps define the others. We often start at the Release Criteria level and work our way “backwards.”