Crispin, Lisa - Agile Testing - A Practical Guide for Testers and Agile Teams

Janet’s Story

In one team I joined, the customers were very picky. In fact, the pickiest I had ever seen. They always asked for a full week of UAT just to be sure they had the time to test it all. They had prepared test cases and checked them all, including all the content, both in English and in French. Showstopper bugs included spelling errors such as a missing accent in the French content. Over time, as they gained more confidence in our releases and found fewer and fewer errors, they relaxed their demands but still wanted a week, just in case they couldn’t get to it right away. Their business group was very busy.

One release came that pushed the timeline. We were being held to the release date but couldn’t get all the functionality in and leave two weeks for the end game. We talked with the business users and we decided to decrease the end game to one week; the business users would perform their UAT while the project team finished up their system testing and cleanup. The only reason we were able to do this was because of the trust the customer had in our team and the consistency of our releases.

The good news was that, once again, the UAT found no issues that could not wait until the next release.

—Janet

Figure 20-1 shows an example timeline with a normal UAT at the end of the release cycle. The team starts working on the next release, doing release planning, and starts the first iteration with all team members ready to go.

Figure 20-1 Release timeline with UAT

Work with customers so that they understand the process, their role, and what is expected of them. If the UAT is not smooth, then the chances are there will be a high level of support needed. An experienced customer test team may have defined test cases, but most often its testing is ad hoc. Customers may approach their testing as if they were doing their daily job but will probably focus on the new functionality. This is an opportunity to observe how people use the system and to get feedback from them on what works well and what improvements would help them.

Testers can provide support to the customers who are doing the UAT by reviewing tests run and defects logged, and by tracking defects to completion. Both of us have found it helpful to provide customers involved in doing UAT with a report of all of the testing done during development, along with the results. That helps them decide where to focus their own testing.

Alpha/Beta Testing

If you are an organization that distributes software to a large customer base, you may not have a formal UAT. You are much more likely to incorporate alpha or beta testing. Your team will want to get feedback on new features from your real customers, and this is one mechanism for doing so. Alpha testing is early distribution of new versions of software. Because there are likely to be some major bugs, you need to pick your customers wisely. If you choose this method of customer feedback, make sure your customers understand their role. Alpha testing is to get feedback on the features—not to report bugs.

Beta testing is closer to UAT. It is expected that the release is fairly stable and can actually be used. It may not be “ready for prime time” for most customers, but many customers may feel the new features are worth the risk. Customers should understand that it is not a formal release and that you are asking them to test your product and report bugs.

As a tester, it is important to understand how customers view the product, because it may affect how you test. Alpha and beta testing may be the only time you get to interact with end users, so take advantage of the chance to learn how well the product meets their needs.

Post-Development Testing Cycles

If you work in a large organization or are developing a component of a large, complex system, you may need to budget time for testing after development is complete. Sometimes the UAT testing, or the test coordination, isn’t as smooth as it could be, so the timeline stretches out. Test environments that include test versions of all production systems may only be available for small, scheduled windows of time. You may need to coordinate test sessions with teams working on other applications that interact with yours. Whatever the reason, you need extra testing time that does not include the whole development team.

Lisa’s Story

I worked on a team developing components of both internal and external applications for a large telecom client. We could only get access to the complete test environment at scheduled intervals. Releases were also tightly scheduled.

The development team worked in two-week iterations. It could release to the test environment only after every third iteration. At that time, there was a two-week system integration and user acceptance test cycle, followed by the release.

Someone from my team needed to direct the post-development testing phase. Meanwhile, the developers were starting a new iteration with new features, and they needed a tester to help with that effort.

The team had to make a special effort to make sure someone in the tester role followed each release from start to finish. For example, I worked from start to finish on release 1. Shauna took over the tester role as the team started work on the first iteration of release 2, while I was coordinating system testing and UAT on release 1. Shauna stayed as primary tester for release 2, while I assumed that role for release 3.

—Lisa

Figure 20-2 shows an example timeline where the UAT was extended. This could happen for any number of reasons, and the issue may not always be UAT. Most of the team is ready to start working on the next release, but often a tester is still working with customers, completing final testing. Sometimes a programmer will be involved as well. There are a couple of options. If the team is large enough, you can probably start the next release while a couple of team members work with the existing release (Release 2—Alternative 2 in Figure 20-2). If you have a small team, you may need to consider an Iteration 0 with programmers doing refactoring or spikes (experiments) on new functionality so that the tester working with the customer does not get left behind (Release 2—Alternative 1 in Figure 20-2).

Figure 20-2 Release timeline—alternative approach with extended UAT

Be creative in dealing with circumstances imposed on your team by the realities of your project. While plans rarely work as expected, planning ahead can still help you make sure the right people are in place to deliver the product in a timely manner.

Deliverables

In the first section of this chapter we talked about what makes a product. The answer to this will actually depend on the audience: Who is accepting the product, and what are their expectations?

If your customers need to meet SOX (Sarbanes-Oxley)compliance requirements, there will be certain deliverables that are required. For example, one customer Janet has worked with felt test results should be thoroughly documented, and made test results one of their SOX compliance measurement points, while a different customer didn’t measure test results at all. Work with compliance and audit personnel to identify reporting needs as you begin a project.

How much documentation is enough? Janet always asks two questions before answering that question: “Who is it for?” and “What are they using it for?” If there are no adequate answers to those questions, then consider whether the documentation is really needed.