Crispin, Lisa - Agile Testing - A Practical Guide for Testers and Agile Teams

We’ve talked about some disadvantages of record/playback tools, but they’re appropriate in the right situation. You may be using a record/playback tool for a good reason: Maybe your legacy code already has a suite of automated tests created in that tool, your team has a lot of expertise in the tool, or your management wants you to use it for whatever reason. You can use recorded scripts as a starting point, then break the scripts into modules, replace hard-coded data with parameters where appropriate, and assemble tests using the modules as building blocks. Even if you don’t have much programming experience, it’s not hard to identify the blocks of script that should be in a module. Login, for example, is an obvious choice.

Record/playback may also be appropriate for legacy systems that are designed in such a way that makes unit testing difficult and hand-scripting tests from scratch too costly. It’s possible to build a record and playback capability into the application, even a legacy application. With the right design, and the use of some human-readable format for the recorded interaction, it’s even possible to build playback tests before the code is built.

GUI Test Automation: From the Dark Ages to Successful Automation in an Agile Environment

Pierre Veragen, SQA Lead at iLevel by Weyerhaeuser, explains how his team used a tool that provided both record/playback and scripting capability productively in a waterfall environment, and then leveraged it when the company adopted agile development.

Back in our waterfall development days, in 2000, we started doing GUI test automation using a record-playback approach. We quickly accumulated tens of thousands of lines of recorded scripts that didn’t meet our testing needs. When I took over 18 months later, I quickly became convinced that the record-playback approach was for the dinosaurs.

When we had a chance to obtain a new test automation tool at the end of 2003, we carefully evaluated tools with these criteria in mind: record capability to help us understand the scripting language, and the ability to build an object-oriented library to cover most of our needs, including test reporting. At the time, TestPartner from CompuWare fulfilled all of our requirements.

We started using TestPartner on a highly complex, CAD-with-engineering application, built in Visual Basic 6, still using a waterfall process. Before we started automating tests, our releases were quickly followed by one or more patches. We focused our automation efforts toward checking the engineering calculations through the GUI, and later, the actual position of the CAD details. These tests included hundreds of thousands of individual verification points, which could never have been done by hand. Within a year, having added a solid set of manual tests of the user interaction, in addition to our automated tests, we were releasing robust software without the usual follow-up patches. We felt confident about our combination of manual and automated tests, which didn’t include a single line of recorded scripts.

In 2004, our group moved to Visual Basic .NET. I spent several months adapting our TestPartner library to activate .NET controls. In 2006, we adopted an Agile methodology. Building on lessons previously learned in the non-Agile world, we achieved astonishing results with test automation. By the end of 2006, team members were able to produce maintainable GUI test scripts and library components after just a few days of training. At the same time, the team embraced unit testing with NUnit and user acceptance tests with FitNesse.

As of this writing, issues are caught at all three levels of our automated testing: Unit, FitNesse, and GUI. The issues found by each of the three testing tiers are of a different nature. Because everything is automated and triggered automatically, issues are caught really fast, in true Agile fashion. Each part of our test automation is bringing value.

Some people feel resources would be better spent on architecture and design, so that GUI test automation isn’t needed. In our development group, each team made its own decision about whether to automate GUI tests.

In case you decide to use GUI test automation, here’s some advice: Stay away from recorded scripts, invest in maintainability, and minimize the required GUI testing with a good architecture of the application. It is my experience that investing in good GUI test automation practices will always pay off.

Pierre’s advice reflects well how good development practices, especially those followed in agile development projects, apply to automated test development as well as to production code development.

Built-In Record & Playback

Gerard Meszaros, agile coach and author of xUnit Test Patterns [2007], describes a situation where the simplest approach turned out to be record/playback. We’ve mentioned drawbacks to record/playback tools, but if you design your code to support them, they can be the best approach.

I was asked to help a team that was porting a “safety sensitive” application from OS2 to Windows. The business was very concerned about the amount of time it would take it to retest the ported system and the likelihood that the team would miss important bugs. The system was designed to only offer the user valid choices that would not compromise safety. They considered using a test recording tool to record tests on the old system and play them back on the new system, but there were no test recording tools available for both OS2 and Windows that could deal with windows drawn using ASCII characters. After reviewing the architecture of the system, we determined that writing xUnit tests would not be a cost-effective way to test the system because much of the business logic was embedded in the user interface logic, and refactoring the code to separate them would be too risky and time-consuming. Instead, we proposed building a Record & Playback test capability right into the system before we ported it.

Even though the rest of the project was milestone-driven, we developed the built-in test mechanism in a very agile way. Each screen required at least one new hook and sometimes several. We started with the most frequently used screens, adding the necessary hooks to record the user’s actions and the systems responses to them into an XML file. We also added the hooks to play back the XML and determine the test results. Initially, we focused our efforts on proving the concept by hooking only the screens we needed to record and then playing back a simple but realistic test. After everyone was convinced the approach would work, we prioritized the screens with respect to how much benefit it would provide. We implemented the hooks one by one until we could automate a significant portion of the tests. We also built an XSLT stylesheet that would format the XML in a Fit-like way, with green cells indicating acceptable results and red cells indicating a failed test step.

In the meantime, the client was identifying the test scenarios that needed test cases. As we finished enough screens to record a particular test, the client would “acceptance test” our hooks by recording and playing back (still on OS2) the test(s) that were waiting for those hooks. When all of the hooks were in place, we could go ahead and port the code, including the test hooks, from OS2 to Windows. After verifying successful playback on OS2, the client would move the XML test files over to Windows and run them against the ported version of the code. The client found this quite easy to do and was able to record a large number of tests in a relatively short period of time. Because the tests were recording actions and responses in business terms, the tests were fairly easy to understand. The client loved the capability, and still raves about how much effort it saved and how much more confidence it has in the product. “Not only did this save tens of man-years of testing effort, but it even uncovered hidden unknown bugs in the legacy system, which we had considered to be the gold standard.”