Crispin, Lisa - Agile Testing - A Practical Guide for Testers and Agile Teams

Organizing Test Results

Everyone involved with delivering software needs easy access to tests and test results. Another aspect of managing tests is keeping track of what tests are from prior iterations and need to keep passing, versus tests that are driving development in the current iteration and may not be passing yet. A continuous integration and build process runs tests for quick feedback on progress and to catch regression failures. Figure 14-4 shows an example of a test result report that’s understandable at a glance. One test failed, and the cause of the failure is clearly stated.

Figure 14-4 Test results from a home-grown test management tool

If you’re driving development with tests, and some of those tests aren’t passing yet, this shouldn’t fail a build. Some teams, such as Lisa’s, simply keep new tests out of the integration and build process until they pass for the first time. After that, they always need to pass. Other teams use rules in the build process itself to ignore failures from tests written to cover the code currently being developed.

As with any test automation tool, you can solve your test management problems with home-grown, open source, or commercial systems. The same criteria we described in the section on evaluating test tools can be applied to selecting a test management approach.

Test management is yet another area where agile values and principles, together with the whole-team approach, applies. Start simple. Experiment in small steps until you find the right combination of source code control, repositories, and build management that keeps tests and production code in synch. Evaluate your test management approach often, and make sure it accommodates all of the different users of tests. Identify what’s working and what’s missing, and plan tasks or even stories to try another tool or process to fill any gaps. Remember to keep test management lightweight and maintainable so that everyone will use it.

Managing Tests For Feedback

Megan Sumrell, an agile trainer and coach, describes how her team coordinates its build process and tests for optimum feedback.

We create a FitNesse test suite for each sprint. In that suite, we create a subwiki for each user story that holds its tests. As needed, we create a setup and teardown per test or suite. If for some reason we don’t complete a user story in the sprint, then we move the tests to the suite for sprint in which we do complete the story.

We scripted the following rule into our build: If any of the suites from the previous sprint fail, then the build breaks. However, if tests in the current sprint are failing, then do not fail the build.

Each test suite has a lengthy setup process, so when our FitNesse tests started taking longer than 10 minutes to run, our continuous integration build became too slow. We used symbolic links to create a suite of tests that serve as our smoke tests, running as part of our continuous integration build process. We run the complete set of FitNesse tests on a separate machine. We set it up to check the build server every five minutes. If a new build existed, then it would pull the build over and run the whole set of FitNesse tests. When it was done, it would then check the build server again every five minutes and after a new build existed, it would repeat the process.

Megan’s team took advantage of features built into their tools, such as symbolic links to organize FitNesse test suites for different purposes—one for a smoke test, others for complete regression testing. The team members get immediate feedback from the smoke tests, and they’ll know within an hour whether there’s a bug that the smoke tests missed.

Go Get Started

Don’t be afraid to get something—anything—in place, even if it’s somewhat deficient. The most important factor in success is to just get started. Many, if not most, successful teams have started with a poor process but managed to turn an inadequate process into something truly essential to the team’s success, one piece at a time. As with so many aspects of agile testing, improving in tiny increments is the key to success.

If you don’t start somewhere, you’ll never get traction on automation. Get the whole team together and start an experiment. Without the right level of test automation, your team can’t do its best work. You need the right test automation to deliver business value frequently. A year or two from now, you’ll wonder why you thought test automation was so hard.

Summary

In this chapter, we considered how to apply agile values, principles, and practices to develop an automation strategy. We discussed the following subjects related to automation:

Use the agile testing quadrants to help identify where you need test automation, and when you’ll need it.

The test automation pyramid can help your team make the right investments in test automation that will pay off the most.

Apply agile values, principles, and practices to help your team get traction on test automation.

Repetitive tasks, continuous integration and build processes, unit tests, functional tests, load tests, and data creation are all good candidates for automation.

Quadrant 3 tests such as usability testing and exploratory testing may benefit from some automation to set up test scenarios and analyze results, but human instincts, critical thinking, and observation can’t be automated.

A simple, whole-team approach, using iterative feedback, and taking enough time can help you get started on a good solution.

When developing an automation strategy, start with the greatest area of pain, consider a multi-layered approach, and strive for continuously revisiting and improving your strategy rather than achieving perfection from the start.

Consider risk and ROI when deciding what to automate.

Take time to learn by doing; apply agile coding practices to tests.

Decide whether you can simply build inputs in-memory, or whether you need production-style data in a database.

Supply test data that will allow tests to be independent, rerunnable, and as fast as possible.

Take on one tool need at a time, identify your requirements, and decide what type of tool to choose or build that fits your needs.

Use good development practices for test automation, and take time for good test design.

Automated tools need to fit into the team’s development infrastructure.