Crispin, Lisa - Agile Testing - A Practical Guide for Testers and Agile Teams

• Rich Resources: Exploratory testers should develop a large set of tools, techniques, test data, friends, and information sources upon which they can draw. The agile test team members should grow their exploratory resources throughout a project and throughout their careers.

To help you understand a day in the life of an agile exploratory tester, here is a short tester’s story:

I arrived at 8:00 a.m. and reviewed what had happened the night before during automated testing. The previous night’s automated tests found some minor but interesting errors. A password field on a login form had accepted a special character, which should have been rejected by the validation. I created an outline as a starting point for my “attack” (a top-level plan and/or risk list).

As I thought about my “plan of attack,” I sketched a small state model of the problem on a flip chart and showed this to a developer and my team’s customer rep. I designed a test incorporating their suggestions, using some data stress inputs that I expected the validation to reject (a 1 MG file of special characters). I executed my test with my stress input, and, sure enough, the system rejected them as expected. I tried a different data set and the system failed with a buffer overflow in the database. I was learning, and we were on the trail of a potentially serious security bug. As the day went on, I explored different inputs to the password field and worked with the team to get the bug fixed.

You can learn from automated test results as well as from exploratory testing. Each type of testing feeds into the other. Develop a broad range of skills so you’ll be able to identify important issues and write tests to prevent them from reoccurring.

The term exploratory testing was popularized by the “context-driven school” of testing. It’s a highly disciplined activity, and it can be learned. Session-based test management is one method of testing that’s designed to make exploratory testing auditable and measurable [Bach, 2003].

Session-Based Testing

Session-based testing combines accountability and exploratory testing. It gives a framework to a tester’s exploratory testing experience so that they can report results in a consistent way.

Janet’s Story

James Bach [2003] compares exploratory testing to putting together a jigsaw puzzle. When I first read his article with the jigsaw puzzle analogy, exploratory testing made perfect sense to me.

I start a jigsaw puzzle by dumping out all of the pieces of the puzzle and then sorting them into the different colors and edge pieces. Next, I put the edge pieces together, which gives me a framework in which to start. The edge of the jigsaw is analogous both to the mission statement, which helps me focus, and to the time-boxing of a session, which keeps me within certain limits.

Session-based testing is a form of exploratory testing, but it is time-boxed and a bit more structured. I learned about session-based testing from Jonathan Bach and found it gave me the structure I needed to do exploratory testing well. I use the same skills as I do for a jigsaw puzzle: I look for patterns in color or shapes or perhaps something that just doesn’t look right, an anomaly. My thought process can take those patterns and make sense of them, using heuristics I have developed to help me solve a puzzle.

—Janet

Like solving the jigsaw puzzle by putting together the outside pieces first, we can use session-based testing to give us the framework in which we work. In session-based testing, we create a mission or a charter and then time-box our session so we can focus on what’s important. Too often as testers, we can go off track and end up chasing a bug that might or might not be important to what we are currently testing.

Sessions are divided into three kinds of tasks: test design and execution, bug investigation and reporting, and session setup. We measure the time we spend on setup versus actual test execution so that we know where we spend the most time. We can capture results in a consistent manner so that we can report back to the team.

For more information on session-based testing, check the bibliography for work by Jonathan Bach.

Automation and Exploratory Testing

We can combine exploratory testing with test automation as well. Jonathan Kohl, in his article “Man and Machine” [2007], talks about interactive test automation to assist exploratory testing. Use automation to do test set up, data generation, repetitive tasks, or to progress along a workflow to the place you want to start. Then you start using your testing skills and experience to find the really “good” bugs, the insidious ones that otherwise escape attention. You can also use an automated test suite to explore. Just modify it a bit, watch the results as it runs, modify it again, and watch what happens.

An Exploratory Tester

With exploratory testing, each tester has a different approach to a problem, and has a unique style of working. However, there are certain attributes that make for a good exploratory tester. A good tester:

Is systematic, but pursues “smells” (anomalies, pieces that aren’t consistent)

Learns to recognize problems through the use of Oracles (principle or mechanism by which we recognize a problem)

Chooses a theme or role or mission statement to focus testing

Time-boxes sessions and side trips

Thinks about what the expert or novice user would do

Explores together with domain experts

Checks out similar or competitive applications

Exploratory testing helps us learn about the behavior of an application. Testers generally know a lot about the application they’re testing. How do they judge whether the application is usable by users who are less technical or not familiar with it? Usability testing is vital for many software systems. We’ll talk about that in the next section.

Usability Testing

There are two types of usability testing. The first type is the kind that is done up front by the user experience folks, using tools such as wire frames to help drive programming. Those types of tests belong in Quadrant 2. In this section, we’re talking about the kind of usability testing that critiques the product. We use tools such as personas and our intuition to help us look at the product with the end user in mind.

User Needs and Persona Testing

Let’s look at an online shopping example. We think about who will use the site. Will it be people who have shopped online before, or will it be brand new users who have no idea how to proceed? We’re guessing it will be a mixture of both, as well as others. Take the time to ask your marketing group to get the demographics of the end users. The numbers might help you plan your testing.

One approach to using personas is for your team to invent several different users of your application representing different experience levels and needs. For our Internet retail application, we might have the following personas:

Nancy Newbie, a senior citizen who is new to Internet shopping and nervous about identity theft