Crispin, Lisa - Agile Testing - A Practical Guide for Testers and Agile Teams

See the bibliography for links to build automation tools and books with more information about improving the build process.

Unit and Component Tests

We can’t overemphasize the importance of automating the unit tests. If your programmers are using TDD as a mechanism to write their tests, then they are not only creating a great regression suite, but they are using them to design high-quality, robust code. If your team is not automating unit tests, its chances of long-term success are slim. Make unit-level test automation and continuous integration your first priority.

Chapter 7, “Technology-Facing Tests that Support the Team,” goes into detail about some of the tools that can be used.

API or Web Services Testing

Testing an API or web services application is easiest using some form of automation. Janet has been on teams that have successfully used Ruby to read in a spreadsheet with all of the permutations and combinations of input variables and compare the outputs with the expected results stored in the spreadsheets. These data-driven tests are easy to write and maintain.

One customer of Janet’s used Ruby’s IRB (Interactive Ruby Shell) feature to test the web services for acceptance tests. The team was willing to share its scripts with the customer team, but the business testers preferred to watch to see what happened if inputs were changed on the fly. Running tests interactively in a semiautomated manner allowed that.

Testing behind the GUI

Testing behind the GUI is easier to automate than testing the GUI itself. Because the tests aren’t affected by changes to the presentation layer and work on more stable business logic code, they’re more stable. Tools for this type of testing typically provide for writing tests in a declarative format, using tables or spreadsheets. The fixtures that get the production code to operate on the test inputs and return the results can generally be written quickly. This is a prime area for writing business-facing tests, understandable to both customers and developers that drive development.

See Chapter 9, “Toolkit for Business-Facing Tests that Support the Team,” for specific tool examples.

Testing the GUI

Even a thin GUI with little or no business logic needs to be tested. The fast pace of agile development, delivering new functionality each iteration, mandates some automated regression tests at the GUI level for most projects. Tool selection is key for successful GUI automation. The automated scripts need to be flexible and easy to maintain. Janet has used Ruby and Watir very successfully when the framework was developed using good coding practices, just as if it were a production application. Time was put into developing the libraries so that there was not a lot of rework or duplication in the code, and changes needed could be made in one place. Making the code easy to maintain increased the ROI on these tests.

A point about testability here—make sure the programmers name their objects or assign IDs to them. If they rely on system-generated identifiers, then every time a new object is added to the page, the IDs will change, requiring changes to the tests.

See Chapter 9, “Toolkit for Business-Facing Tests that Support the Team,” for examples of GUI test frameworks.

Keep the tests to just the actual interface. Check things like making sure the buttons really work and do what they are supposed to. Don’t try to try to test business functionality. Other types of tests that can be automated easily are link checkers. There is no need for someone to manually go through every link on every page to make sure they hit the right page. Look for the low-hanging fruit, automate the things that are simple to automate first, and you’ll have more time for the bigger challenges.

Load Tests

Some types of testing can’t be done without automation. Manual load tests aren’t usually feasible or accurate, although we’ve all tried it at one time or another. Performance testing requires both monitoring tools and a way to drive actions in the system under test. You can’t generate a high-volume attack to verify whether a website can be hacked or can handle a large load without some tool framework.

See Chapter 11, “Critiquing the Product Using Technology-Facing Tests,” for examples of load test automation tools.

Comparisons

Visually checking an ASCII file output by a system process is much easier if you first parse the file and display it in a human-readable format. A script to compare output files to make sure no unintentional changes were made is a lot faster and more accurate than trying to compare them manually. File comparison tools abound, ranging from the free diff to proprietary tools such as WinDiff. Source code management tools, and IDEs have their own built-in comparison tools. These are essential items in every tester’s toolbox. Don’t forget about creating scripts for comparing database tables when doing testing for your data warehouse or data migration projects.

Read more about source code management tools and IDEs in Chapter 7, “Technology-Facing Tests that Support the Team.”

Repetitive Tasks

As we work with our customers to better understand the business and learn what’s valuable to them, we might see opportunities to automate some of their tasks. Lisa’s company needed to mail several forms with a cover letter to all of their clients. The programmers could not only generate the forms but could also concatenate them with the cover letter and greatly speed up the mailing effort. Lisa’s fellow tester, Mike Busse, wrote a spreadsheet macro to do complex calculations for allocating funds that the retirement plan administrators had been doing manually. A lot of manual checklists can be replaced with an automated script. Automation isn’t just for testing.

Data Creation or Setup

Another useful area for automation is data creation or setup. If you are constantly setting up your data, automate the process. Often, we need to repeat something multiple times to be able to recreate a bug. If that can be automated, you will be guaranteed to have the same results each time.

Lisa’s Story

Many of our test schemas, including the ones used for automated regression suites, use canonical data. This canonical or “seed” data was originally taken from production. Some tables in the database, such as lookup tables, don’t change, so they never need to be refreshed with a new copy. Other tables, such as those containing retirement plan, employee, and transaction information, need to start from Ground Zero whenever a regression suite runs.

Our database developer wrote a stored procedure to refresh each test schema from the “seed” schema. We testers may specify the tables we want refreshed in a special table called REFRESH_TABLE_LIST. We have an ant target for each test schema to run the stored procedure that refreshes the data. The automated builds use this target, but we use it ourselves whenever we want to clean up our test schema and start over.

Many of our regression tests create their own data on top of the “seed” data. Our Watir tests create all of the data they need and include logic that makes them rerunnable no matter what data is present. For example, the script that tests an employee requesting a loan from his or her retirement plan first cancels any existing loans so a new one can be taken out.

FitNesse tests that test the database layer also create their own data. We use a special schema where we have removed most constraints, so we don’t have to add every column of every table. The tests only add the data that’s pertinent to the functionality being tested. Each test tears down the data it created, so subsequent tests aren’t affected, and each test is independent and rerunnable.