Crispin, Lisa - Agile Testing - A Practical Guide for Testers and Agile Teams

As of this writing, better functional test tools and IDEs are emerging. These facilitate test maintenance tasks with features such as global search/replace. Twist is an example of a tool implemented as a collection of plug-ins to the Eclipse IDE, so it can take advantage of powerful editing and refactoring features.

Agile-Friendly Tools

Elisabeth Hendrickson [2008] lists some characteristics of effective agile test automation tools. These tools should:

Support starting the test automation effort immediately, using a test-first approach

Separate the essence of the test from the implementation details

Support and encourage good programming practices for the code portion of the test automation

Support writing test automation code using real languages, with real IDEs

Foster collaboration

Part III, “Using Agile Testing Quadrants,” and particularly Chapter 9, “Toolkit for Business-Facing Tests that Support the Team,” contain examples of test automation tools that work well on agile projects.

Implementing Automation

While you’re evaluating tools, think about how quickly your top priority automation need must be addressed. Where will you get the support to help implement it? What training does the team need, and how much time will be available to devote to it? How quickly do you have to ramp up on this tool?

Keep all of these constraints in mind when you’re looking at tools. You might have to settle for a less robust tool than you really want in order to get vital automation going in the short term. Remember that nothing’s permanent. You can build your automation effort step-by-step. Many teams experience unsuccessful attempts before finding the right combination of tools, skills, and infrastructure.

Selenium at Work

Joe Yakich, a software engineer with test automation experience, describes how a team he worked with implemented a test automation effort with Selenium, an open source test automation tool.

The software company I worked for—let’s call it XYZ Corp—had a problem. The product, an enterprise-level web-based application, was a powerful, mature offering. Development projects were managed using Agile and Scrum, and a talented stable of engineers churned out new features quickly. The company was growing steadily.

So, what was the problem? XYZ was facing a future where software testing efforts might not be able to keep pace with the development effort. Software quality issues might slow adoption of the product or—worse yet—cause existing customers to look elsewhere.

Test automation seemed like an obvious way to mitigate these risks, and XYZ was fully aware of it. In fact, they had attempted to create a test automation suite twice before, and failed.

The third time, XYZ chose to use Selenium RC, driven by the Ruby programming language. Selenium RC—the RC is for “Remote Control”—is a tool for test automation. Selenium RC consists of a server component and client libraries. The Java server component acts as an HTTP proxy, making the Selenium Core JavaScript appear to originate from the webserver of the application under test (AUT). The server can start and stop browser sessions (supported browsers include nearly all modern browsers, including Internet Explorer, Firefox, and Safari) and interpret commands to interact with elements such as buttons, links, and input fields. The client libraries allow test scripts to be written in Java, .NET, Perl, Python, and Ruby.

Our team chose Ruby because it’s a purely object-oriented, dynamic, interpreted language with a syntax that is elegant, expressive, and tersely powerful. Most importantly, Ruby is an ideal tool for the creation of a Domain Specific Language (DSL). Ruby is malleable enough for the programmer to first choose the structure and syntax of the DSL and then craft an implementation, as opposed to a more rigid language that might impose constraints on that freedom. One of our goals was to create an automation framework—a DSL—hiding complex detail. We wanted to be able to say things like

editor.save

in our tests instead of

s.click("//table[@class='edit']/tbody/tr[0]//img[@src='save.gif']")

Not only is the former more readable, it’s also far more maintainable. The XPath expression in the latter can be put in a library method to be called as needed. Using a DSL that employs the nouns and verbs of the application allows an engineer writing a test to focus on the test, not the underlying complexity of interacting with on-screen controls.

XYZ created an automation team to build the framework and tests. Creating the framework itself was a time-consuming, technically challenging task. Some of the framework classes themselves were complicated enough to warrant unit tests of their own. After a sufficient amount of test framework was constructed, we began work on actual application tests, using the Ruby RSpec library. RSpec is itself a DSL for test specifications. One of its strengths is the use of simple declarative statements to describe behavior and expectations. One might, for example, write a test using the statement

“ A user should be able to save data in an editor by clicking Save ”

filling in the body of the test with calls to the Selenium-based test framework we had created.

Nearly a year later, we had automated nearly two thousand test cases. Although the majority of the application was covered by automation, other portions of the application required manual testing—we had been forced to make choices and prioritize our efforts. Every week the test suite took longer to run than the preceding week; it now took nearly six hours to complete, and we had begun to think about running tests in parallel. We had not yet managed to expand our testing across all of the browsers supported by the application. The enthusiasm that automation generated had waned somewhat, and we found it necessary to carefully manage expectations, both with upper management and with other engineers. Despite these issues, Selenium was a clear win, for had we not invested heavily in test automation, testing at XYZ would have required hiring an army of test engineers (which would have been prohibitively expensive even had we been able to find enough qualified applicants).

Not everything can be automated, because of budgetary or technical reasons. In addition, exploratory testing is invaluable and should not be neglected. It should be noted, however, that these drawbacks are shared by every other test automation tool currently available, and most of the other automation tools that can rival Selenium’s automation prowess are commercial products that cannot match its price: free.

Good development practices are key to any automation effort. Use an object-oriented approach. As you build your library of test objects, adding new tests becomes easier. A domain specific language helps make business-facing tests understandable to customers, while lowering the costs of writing and maintaining automated test scripts.

Good object-oriented design isn’t the only key to building a suite of maintainable automated tests that pay off. You also need to run the tests often enough to get the feedback your team needs.. Whatever tools we choose must be integrated with our build process. Easy-to-interpret results should come to us automatically.

The tools we choose have to work on our platforms, and must share and play well with our other tools. We have to continually tweak them to help with our current issues. Is the build breaking every day? Maybe we need to hook our results up to an actual traffic light to build team awareness of its status. Did a business-facing test fail? It should be plain exactly what failed, and where. We don’t have extra time to spend isolating problems.