Crispin, Lisa - Agile Testing - A Practical Guide for Testers and Agile Teams

Manual Processes Are Error Prone

Manual testing gets repetitive, especially if you’re following scripted tests, and manual tests get boring very quickly. It’s way too easy to make mistakes and overlook even simple bugs. Steps and even entire tests will be skipped. If the team’s facing a tight deadline, there’s a temptation to cut corners, and the result is a missed problem.

Because manual testing is slow, you might still be testing at midnight on the last day of the iteration. How many bugs will you notice then?

Automated builds, deployment, version control, and monitoring also go a long way toward mitigating risk and making your development process more consistent. Automating these scripted tests eliminate the possibility of errors, because each test is done exactly the same way every time.

The adage of “build once, deploy to many” is a tester’s dream come true. Automation of the build and deploy processes allow you to know exactly what you are testing on any given environment.

Automation Frees People to Do Their Best Work

Writing code test-first helps programmers understand requirements and design code accordingly. Having continual builds run all of the unit tests and the functional regression tests means more time to do interesting exploratory testing. Automating the setup for exploratory tests means even more time to probe into potentially weak parts of the system. Because you didn’t spend time executing tedious manual scripts, you have the energy to do a good job, thinking of different scenarios and learning more about how the application works.

If we’re thinking constantly about how to automate tests for a fix or new feature, we’re more likely to think of testability and a quality design rather than a quick hack that might prove fragile. That means better code and better tests.

Automating tests can actually help with consistency across the application.

Janet’s Story

Jason (one of my fellow testers) and I were working on some GUI automation scripts using Ruby and Watir, and were adding constants for button names for the tests. We quickly realized that the buttons on each page were not consistently named. We were able to get them changed and resolved those consistency issues very quickly, and had an easy way to enforce the naming conventions.

—Janet

See Chapter 9, “Toolkit for Business-Facing Tests that Support the Team,” Chapter 12, “Summary of Testing Quadrants,” and Chapter 14, “An Agile Test Automation Strategy,” for more information about Ruby and Watir.

Books such as Pragmatic Project Automation [2004] can guide you in automating daily development chores and free your team for important activities such as exploratory testing.

Giving Testers Better Work

Chris McMahon described the benefits he’s experienced due to regression test automation in a posting to the agile-testing mailing list in November 2007:

Our UI regression test automation has grown 500% since April [of 2007]. This allows us to focus the attention of real human beings on more interesting testing.

Chris went on to explain, “Now that we have a lot of automation, we have the leisure to really think about what human tests need doing. For any testing that isn’t trivial, we have just about institutionalized a test-idea brainstorming session before beginning execution.” Usually, Chris and his teammates pair either two testers or one tester and a developer. Sometimes a tester generates ideas and gets them reviewed, via a mindmap, a wiki page, or a list in the release notes. Chris observed, “We almost always come up with good test ideas by pairing that wouldn’t have been found by either individual independently.”

Referring to their frequent releases of significant features, Chris says, “Thanks to the good test automation, we have the time to invest in making certain that the whole product is attractive and functional for real people. Without the automation, testing this product would be both boring and stupid. As it is, we testers have significant and interesting work to do for each release.”

We agree with Chris that the most exciting part of test automation is the way it expands our ability to improve the product through innovative exploratory testing.

Projects succeed when good people are free to do their best work. Automating tests appropriately makes that happen. Automated regression tests that detect changes to existing functionality and provide immediate feedback are a primary component of this.

Automated Regression Tests Provide a Safety Net

Most practitioners who’ve been in the software business for a few years know the feeling of dread when they’re faced with fixing a bug or implementing a new feature in poorly designed code that isn’t covered by automated tests. Squeeze one end of the balloon here and another part of it bulges out. Will it break?

Knowing the code has sufficient coverage by automated regression tests gives a great feeling of confidence. Sure, a change might produce an unexpected effect, but we’ll know about it within a matter of minutes if it’s at the unit level, or hours if at a higher functional level. Making the change test-first means thinking through the changed behavior before writing the code and writing a test to verify it, which adds to that confidence.

Janet’s Story

I recently had a conversation with one of the testers on my team who questioned the value of automated tests. My first answer was “It’s a safety net” for the team. However, he challenged that premise. Don’t we just become reliant on the tests rather than fixing the root cause of the problem?

It made me think a bit more about my answer. He was right in one sense; if we become complacent about our testing challenges and depend solely on automated tests to find our issues, and then just fix them enough for the test to pass, we do ourselves a disservice.

However, if we use the tests to identify problem areas and fix them the right way or refactor as needed, then we are using the safety net of automation in the right way. Automation is critical to the success of an agile project, especially as the application grows in size.

—Janet

When they don’t have an automated suite of tests acting as a safety net, the programmers may start viewing the testers themselves as a safety net. It’s easy to imagine that Joe Programmer’s thought process goes like this: “I ought to go back and add some automated unit tests for formatEmployeeInfo, but I know Susie Tester is going to check every page where it’s used manually. She’ll see if anything is off, so I’d just be duplicating her effort.”

It’s nice that a programmer would think so highly of the tester’s talents, but Joe is headed down a slippery slope. If he doesn’t automate these unit tests, which other tests might he skip? Susie is going to be awfully busy eyeballing all those pages.

Teams that have good coverage from automated regression tests can make changes to the code fearlessly. They don’t have to wonder, “If I change this formatEmployeeInfo module, will I break something in the user interface?” The tests will tell them right away whether or not they broke anything. They can go lots faster than teams relying exclusively on manual testing.

Automated Tests Give Feedback, Early and Often

After an automated test for a piece of functionality passes, it must continue to pass until the functionality is intentionally changed. When we plan changes in the application, we change the tests to accommodate them. When an automated test fails unexpectedly, a regression defect may have been introduced by a code change. Running an automated suite of tests every time new code is checked in helps ensure that regression bugs will be caught quickly. Quick feedback means the change is still fresh in some programmer’s mind, so troubleshooting will go more quickly than if the bug weren’t found until some testing phase weeks later. Failing fast means bugs are cheaper to fix.