Crispin, Lisa - Agile Testing - A Practical Guide for Testers and Agile Teams

The regression tests themselves must be under some form of version control. It’s best to keep them in the same source code control system as the production code. That way, when you tag the code for production release, the tag also contains all of the versions of the tests that worked with the code. At minimum, keep a daily backup of the test code.

When tests have been added to the regression suite, their purpose changes. They no longer exist to help drive development, and they are not expected to find new bugs. There sole purpose in life is to detect unexpected changes or side effects in the system.

Checking the “Big Picture”

Hopefully, you wrote task cards to test the story in the context of the larger application and regression test other parts of the system to ensure the new story hasn’t had a negative effect. You may have automated some of those end-to-end tests like the example in Chapter 12, “Summary of Testing Quadrants.”

But sometimes, even if you have a large suite of regression tests, manual exploratory testing can be appropriate. The story isn’t “done” until you’ve completed these tasks as well.

Resources

As you start the iteration, make sure that test environments, test data, and test tools are in place to accommodate testing this iteration’s stories. Hopefully you’ve anticipated these needs, but some requirements might only become obvious when you start working on a story. Collaborate with database experts, system administrators, and other team members to set up any additional infrastructure needed.

You may have brought in outside resources for this iteration to help with performance, usability, security, or other forms of testing. Include them in stand-ups and discussions with the customers as needed. Pair with them and help them understand the team’s objectives. This is an opportunity to pick up new skills.

Iteration Metrics

In Chapter 5, “Transitioning Typical Processes,” we talked a bit about the purpose of metrics, but because metrics are critical to understanding how your coding and testing activities are progressing, we’ll delve into them more here. Know what problem you are trying to solve before you start measuring data points and going to all the work of analyzing the results. In this section, we’ll cover some of the typical measurements that teams gather through the iteration.

Chapter 15, “Tester Activities in Release or Theme Planning,” talks about useful metrics to keep.

Measuring Progress

You need some way to know how much work your team has completed at any point in the iteration and an idea of how much work is left to do. You need to know when it becomes obvious that some stories can’t be completed and the team needs a Plan B. Iteration burndown charts and estimated versus actual time for tasks are examples used to measure team progress. They may or may not provide value for your particular team.

Story or task boards are a good visual way to know the iteration’s status, especially if color coding is used. If too many test task cards are still in the “to do” column or not enough coding task cards have been moved to “Done” or “Tested,” it’s time for the team to think of ways to make sure all of the testing is completed. Maybe some team members need to stop coding and start taking on testing tasks, or maybe one story or a less critical part of a story needs to be put off until the next iteration so that testing for all the other stories can be finished.

This can be accomplished with virtual story boards as well as physical ones. Get creative with your visual effects so that problems are instantly visible. Remember that no story is “done” until it’s tested at all appropriate levels. Teams may have other criteria for when a story is “done,” such as whether it has been peer reviewed or the automated regression tests are completed. On the story board shown in Figure 18-6, the “Done” column for each story row is the rightmost column. The column just to the left of it is the “Verify” column. The story isn’t considered “done” until all the cards, including testing task cards, are in that “Done” column. A glance at the board is enough to know which stories are finished.

Figure 18-6 Story board showing iteration stories and tasks. Used with permission of Mike Thomas. Copyright 2008.

Even teams that don’t track burndown at the task level can do so at the story level. Knowing how much work the team can do each iteration (its velocity) helps with the overall release plan, and the reprioritizing for each iteration. It simply may be enough to know the number of stories completed in an iteration if they tend to average out to the same size. Although plans are tentative at best, it’s helpful to get an idea of about how many stories can be completed by a hard release date or what stories might get done in the upcoming quarter.

Defect Metrics

We talked about defect metrics in Chapter 15, “Tester Activities in Release or Theme Planning” giving you some high level ideas about what to track. Gathering metrics on defects can be very time consuming so always consider the goal before you start to measure. What is the purpose of the metrics you would like to gather? How long will you need to follow the trend before you know if they are useful?

Defect containment is always a favorite metric to capture. When was the defect found? In traditional projects, it is much easier as you have “hard” requirements and coding phases. When the whole team is responsible for quality, and everyone is working together throughout, it is much harder to determine “when” the defect was injected into the system.

We would like to challenge the idea of this type of metric as not necessary in agile development. However, if you find a lot of bugs are slipping through, you may want to start tracking what type of bugs they are so you can address the root cause. For example, if the bugs could have been caught with unit tests, then maybe the programmers need more training on writing unit tests. If the bugs are missed or misunderstood requirements, then maybe not enough time is spent in iteration planning, or acceptance tests aren’t detailed enough.

If you are practicing zero tolerance for defects, then you probably have no need to be tracking defects during coding and testing. A simple card on the story board will give you all the information you need.

Whatever metrics you choose to measure, go for simplicity.

Janet’s Story

In one organization I was with, we tracked the number of defects logged in the DTS over several releases. These were defects that escaped the iteration or were found in the legacy system.Figure 18-7 shows the trend over a year and a half.

Figure 18-7 Sample Defect Trend (but stopped after a while)

At the beginning, the number of issues found right after it was released to QA for final testing was high (33 issues found in one month). The customers found even more issues during UAT which lasted over two months because they were not confident in the quality of the release. In the month that zero defects were reported, we were just starting a new release so there was no new functionality to test. Over the next year, fewer and fewer defects were logged and it becomes impossible to tell where an actual release happened by just looking at the trend.