Crispin, Lisa - Agile Testing - A Practical Guide for Testers and Agile Teams

We talk about some of the metrics you can use later in this chapter.

Test Plan Alternatives

We’ve talked about why to test plan and what you should consider. Now we talk about some of the alternatives to the heavy test plans you may be used to. Whatever type of test plan your organization uses, make it yours. Use it in a way that benefits your team, and make sure you meet your customer’s needs. As with any document your team produces, it should fulfill a purpose.

Lightweight Test Plans

If your organization or customer insists on a test plan for SOX compliance or other regulatory needs, consider a lightweight test plan that covers the necessities but not any extras. Do not repeat items that have already been included in the Project Plan or Project Charter. A sample Test Plan might look something like the one shown in Figure 15-6.

Figure 15-6 Sample Test Plan

A test plan should not cover every eventuality or every story, and it is not meant to address traceability. It should be a tool to help you think about testing risks to your project. It should not replace face-to-face conversation with your customer or the rest of your team.

Using a Test Matrix

Janet uses release planning to work with the testers and customers to develop a high-level test matrix. A test matrix is a simple way to communicate the big picture concerning what functionality you want to test. It gives your team a quick overview of the testing required.

A test matrix is just a list of functionality down the side and test conditions across the top. When thinking about test conditions and functionality, consider the whole application and any impact the new or changed functionality might have on the rest of the application. Testers sitting with customers and thinking about test conditions is what is important.

It can also be a mechanism to track coverage and can be as detailed as you like. A high-level test matrix can be used by the team to show the customer team or management what has been tested already and what is left. A more detailed test matrix can be used by the team to show what is planned for testing and track the progress of the testing. After the matrix has been created, it becomes easy to fill in the squares when testing is done. Keep it simple. Because we like big visible charts that are easy to read, we recommend colors that mean something to your team. For example, green (G) means testing is done and the team is happy with it, while yellow (Y) might mean some testing has been done but more exploratory testing is needed if there is time. Red (R) means something is broken. A white square means it hasn’t been tested yet, and a gray (not applicable) square means it doesn’t need to be tested.

Let’s look at an example. We have a small release we want to put out that calculates shipping costs. In Figure 15-7, different pieces of functionality are represented on one axis, and properties of the shipment are represented on the other. Individual cells are color-coded to show which cases are tested and which need more attention. All of the cells for “<= 2 lbs” are finished, the top three cells for > 4 lbs are done but need more exploratory testing, and the “Ship to Alaska”/“>4 lbs” cell denotes a possible issue.

Figure 15-7 A sample test matrix

Janet’s Story

I had an unexpected side effect from using a test matrix in one project I was on. The customers and testers put the test matrix together, and had thought of all affected functionality for the project and the high-level test conditions they would need. As expected, the act of planning brought a lot of issues out that would have been missed until later.

When they hung the matrix on the wall in their team area, Dave, the developer team lead, expressed an interest. One of the testers explained the matrix to him, and I was surprised when he said it was very useful for them as well. Dave said “I didn’t know that this functionality would affect this area. We need to make sure our unit tests touch on this as well.”

Looking back on this, I shouldn’t have been surprised, but I had never had that experience with the programmers before.

—Janet

A test matrix is a very powerful tool and can be used to help address traceability issues if your team has those problems. Think about what makes sense for your team and adapt it for your team and what makes sense to you.

Test Spreadsheet

Janet has also seen a spreadsheet format used with some success. For example, at WestJet, the first tab in a workbook was a high-level list of functionality that existed in the application. For each row, the team determined if the project affected that piece of functionality. If so, they gave a rating of the expected impact. After the impact of the changes had been determined, decisions about test environments, test data, or UAT could then be made.

Tabs were used for risks and assumptions but could be used for anything your team may need. A flexible format such as a spreadsheet means you can tailor it to work for you.

This information can be used in a number of different ways. It can be used to determine where to concentrate your exploratory testing efforts, or maybe to help create a high-level test matrix to make sure you touch on all of the areas during your testing.

A Whiteboard

If your team is informal and has small releases, any kind of documentation may be too much. Sometimes it’s enough to list the risks and assumptions on a whiteboard or on index cards. Janet has used a whiteboard to manage risks, and it worked quite well. If a risk actually became an issue, the result was documented and crossed off. It was easy to add new risks and mitigation strategies, and the list was visible to the whole team. This could also be done on a wiki page.

We cannot stress enough that you need to know your team and its needs.

Automated Test List

Sometimes you may be required to present more information to your customers, such as a list of test cases. If your team has a tool from which you could extract a list of test case names, you could provide this list easily to anyone who needed it. This would present more of a traditional type detailed test plan but wouldn’t be available until after tests were actually written. We don’t recommend spending any time on this because we don’t see added value, but sometimes this list may be required for risk assessment or auditability.

Preparing for Visibility

If your team is just getting started with agile development, make sure you have necessary infrastructure in place for your early iterations. You may change the way you are tracking progress as you go along, and your retrospectives will help you bring these issues to light. If you’re having problems completing the work planned for each iteration, maybe you need more visible charts or visual aids to help you gauge progress and make mid-iteration adjustments. Do your customers have some way to know how the iteration is progressing and which stories are done? Take time before the each iteration to evaluate whether you’re getting the right kind of feedback to keep track of testing.

Tracking Test Tasks and Status

The effective agile teams we know all follow this simple rule: “No story is done until it’s tested.” This rule can be expanded to say that not only must the story be tested, the code must be checked in, it must have automated tests that are run by a continual build process, it must be documented, or whatever your team’s “doneness” criteria are. At any time during an iteration, you need to be able to quickly assess how much testing work remains on each story, and which stories are “done.” Story or task boards are ideal for this purpose, especially if they use color-coding to denote test tasks vs. development and other types of tasks. Cork boards, steel sheets with magnets, poster-sized sticky notes, or whiteboards all work fine. Give each story its own row, and order them by priority. Have columns for “to do,” “work in progress,” “verify,” and “done.”