Crispin, Lisa - Agile Testing - A Practical Guide for Testers and Agile Teams

—Lisa

Allow yourself to succeed. Work at a sustainable pace. Take the time to refactor as you go or you’ll end up with a mess eventually. As testers, we always have many different tasks to do. If you’re learning a new tool or trying to automate new tests, don’t multitask. Find a big block of time and focus. This is hard, but switching gears constantly is harder.

If business stakeholders are impatient for your team to “just get it done,” analyze the problem with them. What are the risks? How much will a production problem cost? What are the benefits of releasing a quick hack? How much technical debt will it add? What’s the long-term return on investment of a solid design supported with automated tests? How will each approach affect company profitability and customer satisfaction? What about the intangible costs, such as the effect that doing poor-quality work has on team morale? Sometimes the business will be right, but we’re betting that you’ll usually find that up-front investment pays off.

Learn by Doing

Everyone learns in different ways, but when you’ve decided how you’re going to automate a test, jump in and start doing it. In Everyday Scripting with Ruby for Teams, Testers, and You [2007], Brian Marick advises to learn to program by writing a program. Make mistakes! The more problems you have, the more you’ll learn. Getting someone to pair with you will help speed up learning, even if neither one of you is familiar with the tool or the language.

If you don’t have anyone to pair with, talk to the “rubber ducky”: Imagine you’re describing the problem to a coworker. The process of explaining can often make the cause of the problem jump into view. Simply reading a test aloud to yourself can help you find the weaknesses in it.

Apply Agile Coding Practices to Tests

Tests are just as valuable as production code. In fact, production code isn’t much without tests to support it. Treat your tests the same way you treat all code. Keep it in the same source code control tool as your production code. You should always be able to identify the versions of test scripts that go with a particular version of code.

Pairing, refactoring, simple design, modular and object-oriented design, good standards, keeping tests as independent as possible—all of the qualities of good code are also qualities of good automated tests. Agile development is sometimes perceived by the uninformed to be chaotic or lax, when in fact it is highly disciplined. Undertake your automation tasks with the greatest discipline, proceeding in small steps, checking in each step that succeeds. If you’re programming automated scripts, write them test-first, just as any agile programmer would write production code. Keep simplicity in mind, though. Don’t write fancy test scripts with lots of logic unless there’s a good ROI. Those tests need testing and cost more to maintain. Specify tests when you can instead of coding them, and always go with the simplest approach possible.

We can’t emphasize it enough: Test automation is a team effort. The varying experience, skills, and perspectives of different team members can work together to come up with the best approach to automation. Innovate—be creative. Do what works for your unique situation, no matter what the “common wisdom” says.

Automation tools are just one piece of the puzzle. Test environments and test data are essential components. Let’s look at test data next.

Supplying Data for Tests

No matter what tool we use to automate tests, the tests need data to process. Ideally, they need realistic data that matches production data. However, production databases usually contain lots and lots of data, and they can be highly complex. Also, database access slows down tests exponentially. Like so much of agile testing, it’s a balancing act.

Data Generation Tools

As we write this book, there are several cool tools available to generate test data for all kinds of input fields and boundary conditions. Open source and commercial tools such as Data Generator, databene benerator, testgen, Datatect, and Turbo Data are available to generate flat files or generate data directly to database tables. These tools can generate huge varieties of different types of data, such as names and addresses.

It’s also fairly easy to generate test data with a home-grown script, using a scripting language such as Ruby or Python, a tool such a Fit or FitNesse, or a shell script.

Lisa’s Story

Our Watir scripts create randomized test data inputs, both to ensure they are rerunnable (they’re unlikely to create an employee with the same SSN twice), and to provide a variety of data and scenarios. The script that creates new retirement plans produces plans with about 200 different combinations of options. The script that tests taking out a loan randomly generates the frequency, reason, and term of the loan, and verifies that the expected payment is correct.

We have utility scripts to create comma-separated files for testing uploads. For example, there are several places in the system that upload census files with new employee information. If I need a test file with 1,000 new employees with random investment allocations to a retirement plan, I can simply run the script and specify the number of employees, the mutual funds they’re investing in, and the file name. Each record will have a randomly generated Social Security Number, name, address, beneficiaries, salary deferral amounts, and investment fund allocations. Here’s a snippet of the code to generate the investment calculations.

# 33% of the time maximize the number of funds chosen, 33% of the time

# select a single fund, and 33% of the time select from 2-4 funds

fund_hash = case rand( 3)

when 0: a.get_random_allocations(@ fund_list.clone)

when 1: a.get_random_allocations(@ fund_list.clone, 1)

when 2: a.min_percent = 8;

a.get_random_allocations(@ fund_list.clone, rand( 3) + 2)

end

emp['fund_allocations'] = fund_hash_to_string(fund_hash)

Scripts like these have dual uses, both as regression tests that cover a lot of different scenarios and exploratory test tools that create test data and build test scenarios. They aren’t hard to learn to write (see the section “Learning by Doing” earlier in this chapter).

—Lisa

Scripts and tools to generate test data don’t have to be complex. For example, PerlClip simply generates text into the Windows clipboard so it can be pasted in where needed. Any solution that removes enough tedium to let you discover potential issues about the application is worth trying. “The simplest thing that could possibly work” definitely applies to creating data for tests. You want to keep your tests as simple and fast as possible.

Avoid Database Access

Your first choice for testing should try to have tests that can run completely in-memory. They will still need to set up and tear down test data, but the data won’t store in a database. Each test is independent and runs as quickly as any test could. Database access means I/O and disks are inherently slow. Every read to the database slows down your test run. If your goal is to give fast feedback to the team, then you want your tests to run as quickly as possible. A fake object such as an in-memory database lets the test do what it needs to do and still give instant feedback.

Lisa’s Story

One of our build processes runs only unit-level tests, and we try to keep its runtime less than eight minutes, for optimum feedback. The tests substitute fake objects for the real database in most cases. Tests that are actually testing the database layer, such as persisting data to the database, use a small schema with canonical data originally copied from the production database. The data is realistic, but the small amount makes access faster.