Crispin, Lisa - Agile Testing - A Practical Guide for Testers and Agile Teams

Lisa’s Story

As my team built our automation framework one step at a time, we gathered an arsenal of tools. After implementing a continuous build framework with Ant and CruiseControl, we mastered JUnit for unit testing. We knew that unit test automation is the quickest and cheapest way to automate, and provides the fastest feedback to the programmers.

Our legacy system had no automated tests, so we built a GUI regression test suite with Canoo WebTest. This provided good payback because the WebTest scripts were specified, not programmed. They were quick to write and easy to maintain.

After JUnit and WebTest were in place, we experimented with FitNesse and found it worked well for functional testing behind the GUI. We found automating with FitNesse to go relatively quickly. Although FitNesse tests are significantly more expensive to produce and maintain than unit tests, their value in driving development and promoting collaboration among customers, programmers, and testers kept the ROI high.

All of these tools were easy to learn, implement, and integrate with the build process, and provided continual feedback about our regression issues. They were important considerations when we were deciding on our test automation strategy.

—Lisa

When evaluating the payback of your automation efforts, consider less tangible aspects such as whether the tool promoted collaboration between the technical and customer teams. A primary reason to write tests is to help guide development. If the process of writing your automated acceptance tests results in a thorough understanding of business requirements, that’s plenty of payback, even if the tests never find a single regression bug later on.

Think about Test Design and Maintenance

Think about all of the manual test scripts you’ve written in your life. Don’t you just wish those all would have been automated? Wouldn’t your life have been a lot easier? We believe that all scripted tests should be automated. Let’s get started converting those manual scripted tests.

After you get started, it can be quite easy to automate tests. For example, when you have a working FitNesse fixture, adding more test cases requires little effort. This is great when you have a lot of different permutations to test. You’ll probably test more conditions than you would if all your testing was done manually. When Lisa’s team members rewrote their retirement plan’s loan system, they could test hundreds of different possibilities for loan payment processing via FitNesse tests. What happens when three loan payments are processed on the same day? If someone doesn’t make any payments for three months, and then sends in a large payment, is the interest calculated and applied correctly? It was easy to write automated tests to find out.

That’s a great advantage, but it has a down side. Now the team has dozens, or even hundreds, of test cases to maintain. What if the rules about calculating the amount of interest for a loan payment change a year from now? This could require updating every test. If your test tool doesn’t easily accommodate making changes to existing tests, your big suite of automated tests can turn into a headache.

End-to-end tests are particularly tricky to automate because they have the most potential to need maintenance as business rules change. How do we balance the need for automation with the cost?

Test Design

Remember to start with the thin slice or steel thread of the feature you’re testing. Approach automation just as programmers approach coding. Get one small unit of the steel thread working, and then move on to the next. After you’ve covered the whole thin slice, go back and flesh it out.

Chapter 8, “Business-Facing Tests that Support the Team,” explains more about thin slices.

Choose your test pattern thoughtfully. Automate all of the test cases you need, but no more, and automate them at the lowest level that you can. Limit the scope of each test case to one test condition or one business rule. Understand the purpose of the test. Avoid dependencies between tests, because they quickly increase complexity and maintenance expense.

See Chapter 9, “Toolkit for Business-Facing Tests that Support the Team,” for more information about effective test design.

Consider Options

As we’ve mentioned before, the lower the level at which you automate a test, the better the ROI. Push test automation as far down the pyramid as you can. If you have good coverage in your unit and code integration tests, you don’t need to automate as many functional tests. With solid coverage at the lower levels, it might be enough to do end-to-end tests manually to verify the system’s behavior. Use risk analysis to help you decide.

User Interface

The user interface does need to be tested. In some situations, test automation at the GUI level is critical. Perhaps your team is using third-party GUI controls, and you aren’t sure how they will behave. If your risk and ROI analysis supports a lot of automation at the GUI level, make the investment.

If you do automate at the higher levels, don’t go overboard and automate every possible path through the system. You don’t have to keep every automated test created during the development phase in the regression suite; consider tradeoffs of build time and the chance of finding defects. Focus your efforts on covering every important path through the code at the unit, code integration, and functional levels. You’ll get a much better payback.

Strike a Balance

Striking a balance isn’t an agile principle, it’s just common sense. You need a good-enough solution right now, but it doesn’t have to be perfect. Does the tool provide the results you need right now? Does it provide an adequate return on the resources needed to use it for automation? If so, go ahead and use it, and budget time later to look for alternatives. You can improve your automation framework over time. The most important factor is whether your automation tools fit your particular situation right now.

Don’t slide the other way, and think, “OK, we can generate a bunch of scripts with this record tool, get our immediate testing done, and refactor the scripts later to make them maintainable.” While you don’t need to keep searching for the perfectly ideal automation solution, you do need a solution that doesn’t add to your team’s technical debt. Find a balance between “It finds the bugs we need to know about and doesn’t cost too much to maintain” and “This is the most elegant and cool solution we can find.”

Choosing the Right Tools

It’s cool that we have so many tools available to help us solve our automation problems. Don’t go for more sophistication than you need. Lisa’s coworkers have found that a spreadsheet that retrieves data from the database and performs calculations independently of the system is a powerful tool, both for driving development and for verifying the application’s calculations.

We usually minimize test automation at the GUI layer, but there are situations where more GUI automation is appropriate. If the user makes a change at X, what else changes? Some problems only manifest themselves at the GUI level. Lisa tested a bug fix that addressed a back-end problem when retirement plan participants requested a distribution of money from their accounts. The change was surrounded by unit tests, but it was a GUI regression test that failed when the distribution form failed to pop up upon request. Nobody anticipated that a back-end change could affect the GUI, so they probably wouldn’t have bothered to test it manually. That’s why you need GUI regression tests, too.