Crispin, Lisa - Agile Testing - A Practical Guide for Testers and Agile Teams

This scenario tests several processes in the system, including preorder, purchase order receipt, backorder, warehouse cancels, and preorder release. How many Super Tester toys will show as available on the shopping website at the end of all that? While executing the scenario, we’ll probably find other areas we want to investigate; maybe the purchase order application is difficult to use or the warehouse inventory updates aren’t reflected properly in the website. Thinking up and executing these types of tests will teach us more about what our users and other external customers need than running predefined functional tests on narrower areas of the application. As a bonus, it’s fun!

As a tester, we often “make up” test data, but it is usually simple so we can easily check our results. When testing different scenarios, both the data and the flow need to be realistic. Find out if the data comes from another system or if it’s input manually. Get a sample if you can by asking the customers to provide data for testing. Real data will flow through the system and can be checked along the way. In large systems, it will behave differently depending on what decisions are made.

Chapter 14, “An Agile Test Automation Strategy,” examines different approaches to obtaining test data.

Tools to help define the scenarios and workflows can be simple. Data flow or process flow diagrams will help identify some of the common scenarios. These scenarios can help you think through a complex problem if you take the time. Consider the users and their motivation.

Lisa’s Story

Our team planned to rewrite the core functionality of the application that processes the daily buys and sells of mutual funds. These trades are the result of retirement plan participants making contributions, exchanging balances from one fund to another, or withdrawing money from their accounts. Lisa’s coworker, Mike Thomas, studied the existing trade processing flow and diagrammed it so that the team could understand it well before trying to rewrite the code. Figure 10-2 shows a portion of the flow diagram. WT stands for the custodian who does the actual trading. Three different file types are downloaded and translated into readable format: CFM, PRI, and POS. Each of these files feeds into a different part of the application to perform processing and produce various outputs: settled trades, a ticker exception report, and a fund position report.

—Lisa

Figure 10-2 Sample portion of a process flow diagram

When testing end-to-end, make spot checks to make sure the data, status flags, calculations, and so on are behaving as expected. Use flow diagrams and other visual aids to help you understand the functionality. Many organizations depend on reports to make decisions, and those reports seem to be the last thing we verify. If your scenarios have been identified correctly, you might be able to use your application reports to provide a final check.

Exploratory Testing

Exploratory testing (ET) is an important approach to testing in the agile world. As an investigative tool, it’s a critical supplement to the story tests and our automated regression suite. It is a sophisticated, thoughtful approach to testing without a script, and it enables you to go beyond the obvious variations that have already been tested. Exploratory testing combines learning, test design, and test execution into one test approach. We apply heuristics and techniques in a disciplined way so that the “doing” reveals more implications that just thinking about a problem. As you test, you learn more about the system under test and can use that information to help design new tests.

Exploratory testing is not a means of evaluating the software through exhaustive testing. It is meant to add another dimension to your testing. You do just enough to see if the “done” stories are really done to your satisfaction.

The bibliography lists more resources you should investigate to learn more about exploratory testing.

A valuable side effect of exploratory testing is the learning that comes out of it. It reveals areas of the product that could use more automated tests and brings up ideas for new or modified features that lead to new stories.

See the bibliography for some links to more about Rapid Software Testing.

Exploratory Testing Explained

Michael Bolton is a trainer and consultant in rapid and exploratory testing approaches. He teaches a course called Rapid Software Testing, which he co-writes with senior author James Bach. Here’s Michael’s definition of exploratory testing.

Cem Kaner didn’t invent exploratory testing, but he identified and named it in 1983, in the first edition of Testing Computer Software , as an approach all testers use when their brains are engaged in their work. He and James Bach, the other leading advocate of the approach, have long defined exploratory testing as “simultaneous test design, test execution, and learning.” Kaner also defines exploratory testing more explicitly as “a style of testing that emphasizes the freedom and responsibility of the individual tester to continually optimize the value of her work by treating learning, test design, test execution, and test result interpretation as activities that continue in parallel throughout the project.” That’s quite a mouthful. What does it mean?

The most important thing to remember about exploratory testing is that it’s not a test technique on its own. Instead, it’s an approach or a mind-set that can be applied to any test technique. The second thing to remember is that exploratory testing is not merely about test execution; testers can also take an exploratory approach when they’re designing new tests at the beginning of the iteration or analyzing the results of tests that have already been performed. A third important note is that exploratory testing isn’t sloppy or slapdash or unprepared testing. An exploratory approach might require very extensive and elaborate preparation for certain tests—and an exploratory tester’s knowledge and skill set, developed over years, is an often invisible yet important form of preparation. An exploratory test might be performed manually, or might employ extensive use of test automation—that is, any use of tools to support testing. So if exploratory testing isn’t a technique, nor test execution, nor spontaneous, nor manual, what is it that makes a test activity exploratory? The answer lies in the cognitive engagement of the tester—how the tester responds to a situation that is continuously changing.

Suppose that a tester is given the mission to test a configuration dialog for a text editor. A tester using an exploratory approach would use specifications and conversations about the desired behavior to inform test ideas, but would tend to record these ideas in less detail than a tester using a scripted approach. A skilled tester doesn’t generally need much explicit instruction unless the test ideas require some specific actions or data. If so, they might be written down or supplied to a program that could exercise them quickly. Upon seeing the dialog, the exploratory tester would interact with it, usually performing tests in accordance with the original test ideas—but she might also turn her attention to other ideas based on new problems or risks in the dialog as it appeared in front of her. Can two settings conflict in a way not covered by existing tests? The exploratory tester immediately investigates by performing a test on the spot. Does the dialog have a usability issue that could interfere with a user’s work flow? The exploratory tester quickly considers a variety of users and scenarios and evaluates the significance of the problem. Is there a delay upon pressing the OK button? The exploratory tester performs a few more tests to seek a general pattern. Is there a possibility that some configuration options might not be possible on another platform? The exploratory tester notes the need for additional testing and moves on. Upon receiving new builds, the exploratory tester would tend to deemphasize repetition and emphasize variation in order to discover problems missed by older tests that are no longer revealing interesting information. This approach, which has always been fruitful, is even more powerful in environments where the need for repeated testing is handled by the developers’ low-level, automated regression tests.