Crispin, Lisa - Agile Testing - A Practical Guide for Testers and Agile Teams

A very simple example of using Ruby with Watir incorporates the idea of DSL. Methods were created to simplify the tests so that any of the testers could actually create an automated script without knowing any Ruby or Watir.

This next example shows a test, and then two of the methods used in the test.

def test_create_new_user

login 'administrator','admin'

navigate_to_tab 'Manage Users'

click_button "Create New User"

set_text_field "userFirstNameInput", "Ruby"

set_text_field "userLastNameInput", "RubyTester"

click_button "Save Changes"

verify_text "Saved changes"

end

# methods created to support easier test writing

def navigate_to_tab(menuItemName)

@browser.link(:text,menuItemName).click

end

def set_text_field(id, value)

@browser.text_field(:id,id).set value

end

A third level could easily be added if create_new_user was called more than once. Just extract the common code that the test could call:

create_new_user (Ruby, RubyTester)

These tests were well suited to guiding development and providing quick feedback. Making tests easy for testers and customers to write, while keeping the automation framework designed for optimum maintainability, reduced the total cost of ownership of the tests.

—Janet

There are always drawbacks to any tool you use. For example, there are limitations to using objects. Sometimes programmers use custom controls or a new toolkit that your tool might not understand.

Janet’s Story

I started a new job as QA manager, and after much deliberation we decided to drop the vendor tool that the team had been using for a couple of years. We could not figure out what tests were actually being run, or what the real coverage was. We decided to start automating the tests using Ruby and Watir. The automation went fairly quickly at first, but then the tests started failing. We spent a lot of time changing the tests to reflect new object names. The developers were just using the default WebLogic object names, which would change every time a new object was added to the page. The testers went to the developers to ask if they could change the way they were coding. It took a little convincing, but when the developers realized the problems their practice was causing, they changed their habits. Over time, all of the defaults were changed, and each object had an assigned name. The tests became much more robust, and we spent much less time in maintenance mode.

—Janet

Implementing a new test automation tool usually requires some experimentation to get a good balance of testable code and well-designed test scripts. Involving the whole team makes this much easier. Watir is one example of a GUI test tool that we’ve found is well suited to agile projects. Let’s look at a couple more, Selenium and Canoo WebTest.

Selenium.Selenium is another open source tool, actually a suite of tools, for testing web applications. The tests can be written as HTML tables or coded in a number of popular programming languages, and can be run directly in most modern web browsers. A Firefox plug-in called “Selenium IDE” provides a way to learn the tool quickly. A recorder is provided to help create the tests, including writing assertions. Tests can be written in several different common programming and scripting languages, including Java, C#, and Ruby.

See Chapter 14, “An Agile Test Automation Strategy,” for an example of using Selenium RC to create a domain-specific test automation framework.

Canoo WebTest.In WebTest scripts, tests are specified as “steps” in XML files, simulating a user’s actions through a web UI. Here’s an example of how a WebTest script might invoke a page and verify the results:

Rather than driving an actual browser, as Selenium and Watir do, WebTest simulates the desired browser using HtmlUnit. The advantage of specifying tests as opposed to coding test scripts, is because there’s no logic in them, you don’t have to test the test.

Lisa’s Story

My team chose WebTest to automate smoke tests for our legacy application for several reasons. Because the scripts are written in XML, the programmers on the team were comfortable using the tool. It uses Ant to run the tests, so integrating it into the continuous build process was simple. It’s easy to learn, and the tests can be designed in a modular fashion, so they’re fairly easy to maintain. WebTest supports testing PDF files, emails, and Excel files, all of which are widely used in our application.

Being accustomed to powerful commercial test tools, I was skeptical of the concept of specifying tests, as opposed to programming them. I was amazed at how effective the simple tests were at catching regression bugs. It’s possible to put logic into the tests using Groovy or other scripting languages, but we’ve only found the need in a few cases.

Writing a few tests per iteration, I automated smoke tests for all of the critical areas of our application in eight months. These simple tests find regression bugs regularly. We refactor the tests frequently, so they are relatively easy to maintain. Our ROI on these tests has been tremendous.

—Lisa

Selenium, WebTest, and Watir are just three examples of the many open source tools available for GUI testing as of the time we wrote this book. Many teams write their own test automation frameworks. Let’s look at an example in the next section.

“Home-Brewed” Test Automation Tools

Bret Pettichord [2004] coined the term “home-brewed” for the tools agile teams create to meet their own unique testing needs. This allows even more customization than an open source tool. The goal of these tools is usually to provide a way for nontechnical customer team members and testers to write tests that are actually executable by the automated tool. Home-brewed tools are tailored to the exact needs of the project. They can be designed to minimize the total cost of ownership. They’re often built on top of existing open source tools.

Janet has been involved in a few projects that have used Ruby and Watir to create a full framework for functional testing. These frameworks allowed customers to specify tests that were then turned into a functional regression suite.

No test tool guarantees success. In fact, the history of test automation is littered with failed attempts. Having the whole team think about the best tools to use is a big help, but no matter what tool you use, you need a smart approach to writing tests. We’ll discuss that in the next section.

PAS Functional Testing

This next story is about one project Janet worked on that enjoyed success with home-brewed test automation.

PAS is a production accounting application for the oil and gas industry. Using gross meter readings and contract agreements, it must calculate ownership of the various products down to a very precise level (i.e., the components in gas). There are literally thousands of interactions between the combinations available in configuring the system and the actual outputs visible to a user. Given the large number of interactions, PAS has employed many complementary strategies for testing.

Joseph King, one of the initial programmers and agile coach for the team, tells us the story of how they accomplished their functional testing.

At the lowest level, there are developer functional tests that exercise specific functions via an API and verify the results using another read-only user API. There are currently over 24,000 tests implemented in JUnit that every developer must run before they can “check in” their changes to the source code.

The next level is a set of GUI tests that test the marshalling of user data back and forth to the API, particularly around “master-data” creation and updates. There are currently over 500 of these tests implemented using Watij (an open source library similar to Watir but using Java) and JUnit that run multiple times a day.