Crispin, Lisa - Agile Testing - A Practical Guide for Testers and Agile Teams

In Gerard’s story, the team worked together to retrofit testability onto a system that wasn’t designed for testability. They gave their customers a way to capture their test scenarios on one platform and play them back on both platforms to verify the successful port. This is a stellar example of the whole-team approach. When everyone on the team collaborates on a test automation solution, there’s a much better chance it’s going to succeed.

Some agile teams get value from commercial or open source test tools, while others prefer a completely customized approach. Many testers find value writing simple scripts in a scripting language such as Ruby, or a shell, to automate mundane but necessary tasks, generate test data, or drive other tools. Books such as Everyday Scripting with Ruby for Teams, Testers, and You give a roadmap for this approach. If you’re a tester without a strong programming background, we encourage you to pick up a book, find an online tutorial, or take a class on a scripting language, and see how easy it can be to write useful scripts.

See more examples of specific tools for business-facing tests in Chapter 9, “Toolkit for Business-Facing Tests that Support the Team.”

What we’re trying to tell you is that you can use many different tools. Look at the problem you are trying to solve and decide as a team the easiest and most effective way to solve it. Every so often, step back and take a look at the tools you’re using. Is everyone on the team happy with them? Are you missing problems because you don’t have the right tools? Budget time to explore new tools and see if they might fill gaps or replace a tool that isn’t paying off.

If your team is new to agile development, or working on a brand-new project, you might be faced with choosing tools and setting up test environments during the early iterations, when you also might be working on high-risk stories. Don’t expect to be able to deliver much business value if you’re still creating your test infrastructure. Plan in lots of time for evaluating tools, setting up build processes, and experimenting with different test approaches.

Applying Agile Principles to Test Automation

Every team, every project, and every organization has a unique situation with unique automation challenges. Each has its own culture, history, resources, business pressures, products, and experience. No matter what your team’s situation, you can use the agile principles and values discussed in Chapter 2 to help you find solutions. Concepts such as courage, feedback, simplicity, communication, continuous improvement, and responding to change aren’t just agile ideas—they’re qualities that are common to all successful teams.

Keep It Simple

The agile maxim of “do the simplest thing that could possibly work” applies to tests as well as code. Keep the test design simple, keep the scope minimal, and use the simplest tool that will do the job.

Simplicity is a core agile value for a good reason. The best place to start is the simplest approach you can think of. However, doing the simplest thing doesn’t mean doing the easiest thing. It involves really thinking about what you need now and taking baby steps to get there. By keeping things simple, if you do make a bad choice, you won’t go too far off track before realizing the error of your ways.

It’s easy to get involved in a task and slip away from the basics into some intriguing challenge. Weigh the ROI of every automation task before you do it. Automation is fun (when you get past the scary part of getting started). It’s tempting to try something difficult just because you can. Like all other aspects of testing in an agile development project, the only way to keep up is to do only the minimum required.

Use the simplest tool you can get away with. Remember the test pyramid. If a customer-facing test can be most easily automated at the unit level, do it there. Lisa sometimes writes test cases in FitNesse, only to learn the programmers can automate them much faster as JUnit tests. Conversely, sometimes the programmers use FitNesse for TDD instead of JUnit, because the code they’re writing lends itself to testing in one of the FitNesse fixture formats.

Iterative Feedback

Short iterations allow us to experiment with various automation approaches, evaluate results, and change course as quickly as needed. Commit to an automation effort, such as developing a test framework in-house or implementing an open source tool for at least a couple of iterations. After each iteration, look at what’s working and what’s not working. Think of ideas to overcome problems, and try those in the next iteration. If it’s not the right solution, try something else for a few iterations. Don’t get sucked into a quagmire where you’ve put so many resources into a tool, and have so many tests that use it, that you feel you can’t switch tools. Between the many open source and commercial tools, plus programmers’ ability to write home-grown test tools, there’s no reason to settle for less than the optimum tool.

Lisa’s Story

One of my early XP teams struggled to find a good way to automate customer-facing acceptance tests for a Java-based web application. This was back when there were far fewer tool options for agile teams. First, we tried an open source tool that simulated a browser, but it lacked the features we required. It just wasn’t quite robust enough. We discussed this at the next retrospective.

We decided to try using the unit testing tool for testing behind the GUI for the next two iterations. By committing to two iterations, we felt we were giving ourselves enough time to give the tool a good try, but not so much time that we would have too much invested if it weren’t the right solution. The customers found the unit tests hard to read, and there was logic in the GUI we couldn’t test with this tool.

After another discussion during our retrospective, we then committed to two iterations of using a vendor GUI test tool I had used extensively on previous projects. The Java programmers found it slow going because the tool used a proprietary scripting language, but it worked well enough to do the minimum automation needed. After two iterations, we decided that it wasn’t ideal, but at the time there weren’t a lot of other options, and it was the best one we had.

In hindsight, we should have kept looking for a better option. Perhaps we could have developed our own test harness. We were able to automate about 60% of the regression tests above the unit level using the vendor tool, which seemed great at the time. If we had pushed ourselves a little more, we might have done a lot better.

—Lisa

Use iterations to your advantage. They facilitate a step-wise approach. If your idea’s a dud, you’ll know quickly and have a chance to try a different one. Don’t be afraid to keep looking, but don’t keep looking for the perfect solution if one you try performs adequately.

Whole-Team Approach

Agile development can’t work without automation. Fortunately, the whole-team approach, which we explored in Chapter 1, means that a wider range of skills and resources are available to find and implement a useful automation strategy. Attacking the problem as a team means it’s more likely that code will be designed for testability. Programmers, testers, and other team members will collaborate to automate tests, bringing multiple viewpoints and skill sets to the effort.

The whole-team approach helps overcome the fear barrier. Automation tasks can be overwhelming to start with. Knowing there are other people with different skills and experience to help gives us courage. Being able to ask for and receive help gives us confidence that we can achieve adequate coverage with our automated tests.