Crispin, Lisa - Agile Testing - A Practical Guide for Testers and Agile Teams

Consider testability in your code design, and choose your test tools wisely, because they need to work with your code.

We need some way to organize tests so that they can be used effectively and put into version control.

Chapter 10 Business-Facing Tests that Critique the Product

This chapter covers the third quadrant of the testing matrix. In Chapter 8, “Business-Facing Tests that Support the Team,” we talked about the second quadrant and how to use business-facing tests to support programming. In this chapter, we show you how to critique the product with different types of business-facing tests. We’ll also talk about tools that might help with these activities.

Introduction to Quadrant 3

Remember that business-facing tests are those you could describe in terms that would (or should) be of interest to a business expert. When we mention testing in traditional phased approaches, it pretty much always means critiquing the product after it is built. By now, you might think that in agile development this part of testing should be easy. After all, we just spent all that time making sure it works as expected. The requirements have all been tested as they were built, including security and other nonfunctional requirements, right? All that’s left is to possibly find some obscure or interesting bugs.

As testers, we know that people make mistakes. No matter how hard we try to get it right the first time, we sometimes get it wrong. Maybe we used an example that didn’t test what we thought it did. Or maybe we recorded a wrong expected result so the test passed, but it was a false positive. The business expert might have forgotten some things that real users needed. The best customer may not know what she wants (or doesn’t want) until she sees it.

Critiquing or evaluating the product is what testers or business users do when they assess and make judgments about the product. These evaluators form perceptions based on whether they like the way it behaves, the look and feel, or the workflow of new screens. It is easier to see, feel, and touch a product and respond than to imagine what it will look like when it is described to you.

It’s difficult to automate business-facing tests that critique the product, because such testing relies on human intellect, experience, and instinct. However, automated tools can assist with aspects of Quadrant 3 tests (see Figure 10-1), such as test data setup. The last section of this chapter contains examples of the types of tools that help teams focus on the important aspects of evaluating the product’s value.

Figure 10-1 Quadrant 3 tests

While much of the testing we discuss in this chapter is manual, don’t make the mistake of thinking that this manual testing will be enough to produce high-quality software and that you can get away with not automating your regression tests. You won’t have time to do any Quadrant 3 tests if you haven’t automated the tests in Quadrants 1 and 2.

Evaluating or critiquing the product is about manipulating the system under test and trying to recreate actual experiences of the end users. Understanding different business scenarios and workflows helps to make the experience more realistic.

Demonstrations

We recommend showing customers what you’re developing early and often. As soon as a rudimentary UI or report is available during story development, show it to the product owner or other domain expert on the team. However, not everyone on the business side will get a chance to see the iteration’s deliverables until the iteration demo. End-of-iteration demonstrations are an opportunity for the business users and domain experts to see what has been delivered in the iteration and revise their priorities. It gives them a chance to say, “That’s what I said, but it’s not what I meant.” This is a form of critiquing the product.

Janet’s Story

I worked on a project that had five separate teams of eight to ten members, all developing the same system. Even though they were on the same floor, communication was an issue. There were many dependencies and overlaps, so the programmers depended on team-lead meetings to share information. However, the business users and testers needed to see what was being developed by other teams. They relied on end-of-iteration demonstrations given by each team to learn what the other teams were doing.

—Janet

Demonstrations to the executives or upper management can instill confidence in your project as well. One of the downfalls of a phased project is there is nothing to see until the very end, and management has to place all of its trust in the development team’s reports. The incremental and iterative nature of agile development gives you a chance to demonstrate business value as you produce it, even before you release it. A live demonstration can be a very powerful tool if the participants are actively asking questions about the new features.

Rather than waiting until the end of the iteration, you can use any opportunity to demonstrate your changes. A recent project Janet worked on used regularly scheduled meetings with the business users to demonstrate new features in order to get immediate feedback. Any desired changes were fed into the next iteration.

Choose a frequency for your demonstrations that works for your team so that the feedback loop is quick enough for you to incorporate changes into the release.

Chapter 19, “Wrap Up the Iteration,” talks about end-of-iteration demonstrations and reviews.

Informal demos can be even more productive. Sit down with a business expert and show her the story your team is currently coding. Do some exploratory testing together. We’ve heard of teams that get their stakeholders to do some exploratory testing after each iteration demo in order to help them think of refinements and future stories to change or build on the functionality just delivered.

Scenario Testing

Business users can help define plausible scenarios and workflows that can mimic end user behavior. Real-life domain knowledge is critical to creating accurate scenarios. We want to test the system from end to end but not necessarily as a black box.

One good technique for helping the team understand the business and user needs is “soap opera testing,” a term coined by Hans Buwalda [2003]. The idea here is to take a scenario that is based on real life, exaggerate it in a manner similar to the way TV soap operas exaggerate behavior and emotions, and compress it into a quick sequence of events. Think about questions like, “What’s the worst thing that can happen, and how did it happen?”

Soap Opera Test Example

Lisa worked on an Internet retail site, where she found soap opera tests to be effective. Here’s an example of a soap opera scenario to test inventory, preorder, and backorder processes of an Internet retailer’s warehouse.

The most popular toy at our online toy store this holiday season is the Super Tester Action Figure. We have 20 preorders awaiting receipt of the items in our warehouse. Finally, Jane, a warehouse supervisor, receives 100 Super Tester Action figures. She updates the inventory system to show it is available inventory against the purchase order and no longer a preorder. Our website now shows Super Tester Action Figures available for delivery in time for the holidays. The system releases the preorders, which are sent to the warehouse. Meanwhile, Joe, the forklift driver, is distracted by his cell phone, and accidentally crashes into the shelf containing the Super Tester Action Figures. All appear to be smashed up beyond recognition. Jane, horrified, removes the 100 items from available inventory. Meanwhile, more orders for this popular toy have piled up in the system item. Sorting through the debris, Jane and Joe find that 14 of the action figures have actually survived intact. Jane adds them back into the available inventory.