Crispin, Lisa - Agile Testing - A Practical Guide for Testers and Agile Teams

We gave the customer a couple of links to explain what XPath was so that he could explore whether this was a good solution for him. To my amazement, he was delighted with XPath (I now know who to turn to when I have a problem with XPath) and filled in the Fit test.

The interesting bit for me is that as soon as he knew what the message looked like and how it was structured, he realized that it did not really support the business—we were sending information that was outside our scope of our work and that should have been supplied by another system. He was also skeptical about the speed at which the external team could add new products due to the complex nature of the XML.

Most agile people we tell this story to think we have the “perfect customer!”

Even if your customers aren’t perfect, involving them in writing customer tests gives them a chance to identify functionality that’s outside the scope of the story. We try to write customer tests that customers can read and comprehend. Sometimes we set the bar too low. Collaborate with your customers to find a tool and format for writing tests that works for both the customer and development teams.

It’s fine to say that our customers will provide to us the examples that we need to have in order for us to understand the value that each story should deliver. But what if they don’t know how to explain what they want? In the next section, we’ll suggest ways to help customers define their conditions of satisfaction.

Eliciting Requirements

If you’ve ever been a customer requesting a particular software feature, you know how hard it is to articulate exactly what you want. Often, you don’t really know exactly what you want until you can see, feel, touch and use it. We have lots of ways to help our customers get clarity about what they want.

Ask Questions

Start by asking questions. Testers can be especially good at asking a variety of questions because they are conscious of the big picture, the business-facing and technical aspects of the story, and are always thinking of the end user experience. Types of general questions to ask are:

Is this story solving a problem?

If so, what’s the problem we’re trying to solve?

Could we implement a solution that doesn’t solve the problem?

How will the story bring value to the business?

Who are the end users of the feature?

What value will they get out of it?

What will users do right before and right after they use that feature?

How do we know we’re done with this story?

One question Lisa likes to ask is, “What’s the worst thing that could happen?” Worst-case scenarios tend to generate ideas. They also help us consider risk and focus our tests on critical areas. Another good question is, “What’s the best thing that could happen?” This question usually generates our happy path test, but it might also uncover some hidden assumptions.

Use Examples

Most importantly, ask the customer to give you examples of how the feature should work. Let’s say the story is about deleting items out of an online shopping cart. Ask the customer to draw a picture on a whiteboard of how that delete function might look. Do they want any extra features, such as a confirmation step, or a chance to save the item in case they want to retrieve it later? What would they expect to see if the deletion couldn’t be done?

Examples can form the basis for our tests. Our challenge is to capture examples, which might be expressed in the business domain language, as tests that can actually be executed. Some customers are comfortable expressing examples using a test tool such as Fit or FitNesse as long as they can write them in their domain language.

Let’s explore the difference between an example and a test with a simple story (see Figure 8-4). People often get confused between these two terms.

Figure 8-4 Story to use as a base for examples and tests

An example would look something like this:

There are 5 items on a page. I want to select item 1 for $20.25 and put it in the shopping cart. I click to the next page, which has 5 more items. I select a second item on that page for $5.38 and put it in my shopping cart. When I say I’m done shopping, it will show both the item from the first page and the item from the second page in my shopping cart, with the total of $25.63

The test could be quite a bit different. We’ll use a Fit type format in Table 8-1 to show you how the test could be represented.

Table 8-1 Test for Story PA-2

The test captures the example in an executable format. It might not use exactly the same inputs, but it encapsulates the sample user scenario. More test cases can be written to test boundary conditions, edge cases, and other scenarios.

Multiple Viewpoints

Each example or test has one point of view. Different people will write different tests or examples from their unique perspectives. We’d like to capture as many different viewpoints as we can, so think about your users.

Getting the requirements right is an area where team members in many different roles can jump in to help. Business analysts, subject matter experts, programmers, and various members of the customer team all have something to contribute. Think about other stakeholders, such as your production support team. They have a very unique perspective.

We often forget about nonfunctional requirements such as “How long does the system need to be up? What happens if it fails? If we have middleware that passes messages, do we expect messages to be large enough that we might need to consider loss during transmission? Or will they be a constant size? What happens if there is no traffic for hours? Does the system need to warn someone?” Testing for these types of requirements usually falls into quadrants 3 and 4, but we still need to write tests to make sure they get done.

All of the examples that customers give to the team add up quickly. Do we really have to turn all of these into executable tests? Not as long as we have the customers there to tell us if the code is working the way they want. With techniques such as paper prototyping, designs can be tested before a line of code is written.

Wizard of Oz Testing

Gerard Meszaros, a Certified ScrumMaster (Practicing) and Agile Coach, shared his story about Wizard of Oz Testing on Agile Projects. He describes a good example of how artifacts we generate to elicit requirements can help communicate meaning in an unambiguous form.

We thought we were ready to release our software. We had been building it one iteration at a time under the guidance of an on-site customer who had prioritized the functionality based on what he needed to enter into integration testing with his business partners. We consciously deferred the master data maintenance and reporting functionality to later iterations to ensure we had the functionality needed for integration testing ready. The integration testing went fine, with just a few defects logged (all related to missing or misunderstood functionality). In the meantime, we implemented the master data maintenance in parallel with integration testing in the last few iterations. When we went into acceptance testing with the business users, we got a rude shock: They hated the maintenance and reporting functionality! They logged so many defects and “must-have improvements” that we had to delay the release by a month. So much for coming up with a plan that would allow us to deliver early!