Crispin, Lisa - Agile Testing - A Practical Guide for Testers and Agile Teams

The contents and formatting of a report are important, of course, but for online reports, the speed at which they come up is critical too. Our plan administrators wanted complete freedom to specify any date range for some transaction history reports. Our DBA, who coded the reports, warned that for a large company’s retirement plan, data for more than a few months worth of transactions could take several minutes to render. Over time, companies grew, they had more and more transactions, and eventually the user interface started timing out before it could deliver the report. When testing, try out worst-case scenarios, which could eventually become the most common scenario.

—Lisa

If you’re tackling a project that involves lots of reports, don’t give in to the temptation to leave them to the end. Include some reports in each iteration if you can. One report could be a single story or maybe even broken up into a couple of stories. Use mock-ups to help the customers decide on report contents and formatting. Find the “thin slice” or “critical path” in the report, code that first, and show it to your customer before you add the next slice. Incremental development works as well with reports as it does with other software.

See Chapter 8, “Business-Facing Tests that Support the Team,” for information about using thin slices.

Sometimes your customers themselves aren’t sure how a report should look or how to approach it incrementally. And sometimes nobody on the team anticipates how hard the testing effort will prove to be.

Lisa’s Story

Like other financial accounts, retirement plans need to provide periodic statements to account holders that detail all of the money going into and out of the account. These statements show the change in value between the beginning and ending balances and other pertinent information, such as the names of account beneficiaries. Our company wanted to improve the account statements, both as a marketing tool and to reduce the number of calls from account holders who didn’t understand their statements.

We didn’t have access to our direct competitors’ account statements, so the product owner asked for volunteers to bring in account statements from banks and other financial institutions in order to get ideas. Months of discussions and experimentation with mock-ups produced a new statement format, which included data that wasn’t on the report previously, such as performance results for each mutual fund.

Stories for developing the new account statement were distributed throughout two quarters worth of iterations. During the first quarter, stories to collect new data were done. Testing proved much harder than we thought. We used FitNesse tests to verify capturing the different data elements, which lulled us into a false sense of security. It was hard to cover all of the variations, and we missed some with the automated tests. We also didn’t anticipate that the changes to collect new data could have an adverse effect on the data that already displayed on the existing statements.

As a result, we didn’t do adequate manual testing of the account statements. Subtle errors slipped past us. When the job to produce quarterly statements ran, calls started coming in from customers. We had a mad scramble to diagnose and fix the errors in both code and data. The whole project was delayed by a quarter while we figured out better ways to test and added internal checks and better logging to the code.

—Lisa

Short iterations mean that it can be hard to make time for adequate exploratory testing and other Quadrant 3 activities. Let’s look at tools that might help speed up this testing and make time for vital manual and visual tests.

Tools to Assist with Exploratory Testing

Exploratory testing is manual testing. Some of the best testing happens because a person is paying attention to details that often get missed if we are following a script. Intuition is something that we cannot make a machine learn. However, there are many tools that can assist us in our quest for excellence.

Tools shouldn’t replace human interaction; they should enhance the experience. Tools can provide testers with more power to find the hard-to-reproduce bugs that often get filed away because no one can get a handle on them. Exploratory testing is unconventional, so why shouldn’t the tools be as well? Think about low-effort, high-value ways that tools can be incorporated into your testing.

Computers are good at doing repetitive tasks and performing calculations. These are two areas where they are much better than humans, so let’s use them for those tasks. Because testing needs to keep pace with coding, any time advantage we can gain is a bonus.

See the bibliography for references to Jonathan Kohl’s writings on using human and automation power together for optimal testing.

In the next few sections, we’ll look at some areas where automation can leverage exploratory testing. The ones we cover are test setup, test data generation, monitoring, simulators, and emulators.

Test Setup

Let’s think about what we do when we test. We’ve just found a bug, but not one that is easily reproducible. We’re pretty sure it happens as a result of interactions between components. We go back to the beginning and try one scenario after another. Soon we’ve spent the whole day just trying to reproduce this one bug.

Ask yourself how you can make this easier. We’ve found that one of the most time-consuming tasks is the test setup and getting to the right starting point for your actual test. If you use session-based testing, then you already know how much time you spend setting up the test, because you have been tracking that particular time waster. This is an excellent opportunity for some automation.

The tools used for business-facing tests that support the team described in Chapter 9 are also valuable for manual exploratory testing. Automated functional test scripts can be run to set up data and scenarios to launch exploratory testing sessions. Tests configured to accept runtime parameters are particularly powerful for setting up a starting point for evaluating the product.

Lisa’s Story

Our Watir test scripts all accept a number of runtime parameters. When I need a retirement plan with a specific set of options, and specific types of participants, I can kick off a Watir script or two with some variables set on the command line. When the scripts stop, I have a browser session with all of the data I need for testing already set up. This is so fast that I can test permutations I’d never get to using all-manual keystrokes.

—Lisa

The test scripts you use for functional regression testing and for guiding development aren’t the only tools that help take the tedium out of manual exploratory testing. There are other tools to help set up test data as well as to help you evaluate the outputs of your testing sessions.

Whatever tool you are using, think about how it can be adapted to run the scenario over and over with different inputs plugged in. Janet has also successfully used Ruby with Watir to set up tests to run multiple times to help identify bugs. Tools that drive the browser or UI in much the same way that an end user would makes your testing more reliable because you can play it back on your monitor and watch for anything that might not look as it should during the setup. When you get to the place where the test actually starts, you can then use your excellent testing abilities to track down the source of the bug.

Test Data Generation

PerlClip is an example of a tool that you can use to test a text field with different kinds of inputs. James Bach provides it free of charge on his website, www.satisfice.com, and it can be very helpful in validating fields. For example, if you have a field that will accept a maximum input of 200 characters, testing this field and its boundaries manually would be very tedious. Use PerlClip to create a string, put it in your automation library, and have your automation tool call the string to test the value.