Crispin, Lisa - Agile Testing - A Practical Guide for Testers and Agile Teams

These concerns are an essential part of the picture, but still only part of the picture. We need tools that help us devise test environments that mimic production. We need ways to keep these test environments independent, unaffected by changes programmers might be making.

Building test infrastructure can be a big investment, but it’s one our agile team needs to make to get a jump on test automation. Hardware, software, and tools need to be identified and implemented. Depending on your company’s resources, this might be a long-term project. Brainstorm ways to cope in the short term, while you plan how to put together the infrastructure you really need to minimize risk, maximize velocity, and deliver the best possible product.

Managing Automated Tests

Let’s say we need a way to find the test that verifies a particular scenario, to understand what each test does, and to know what part of the application it verifies. Perhaps we need to satisfy an audit requirement for traceability from each requirement to its code and tests. Automated tests need to be maintained and controlled in the same way as production source code. When you tag your production code for release, the tests that verified that functionality need to be part of the tag.

Here’s an example where that comes in handy. We just found a problem in the code under development. Is it a new problem, or has it been lurking in the code for a while and somehow missed by the test? We can deploy the tag that’s in production, try to reproduce the problem, and investigate why the tests didn’t catch it. Lisa’s team recently had a situation where the regression suite missed a bug because a database constraint was missing in the test schema. That kind of problem is hard to pinpoint if you aren’t tying your test code versions to your production code versions.

Organizing Tests

Many tools come with their own means of organization. For example, FitNesse comes with its own wiki, with a hierarchical organization, and built-in version control. As of this writing, FitNesse is starting to provide support for source code control tools such as Subversion. Scripts written in other test tools, such as Watir and Canoo WebTest, can and should be maintained within the same source code control system as production code, just as the unit tests are.

Organizing Tests with the Project under Test

We asked some agile testing experts how they manage tests. Dierk König, founder and project manager of Canoo WebTest, explained how his teams have managed their automated tests to satisfy the needs of both the development and customer teams.

We always organize our tests alongside the project under test. That is, test sources are stored together with the project sources in the exact same repository, using the same mechanisms for revision control, tagging, and sharing the test base.

WebTest comes with a standard layout of how to organize tests and test data in directories. You can adapt this to any structure you fancy, but the “convention over configuration” shows its strength here. In large projects, every sub-project maintains its own test base in a “webtest” subdirectory that follows the convention.

Whenever a client did not follow this approach, the experience was very painful for all people involved. We have seen huge databases of test descriptions that did not even feature a proper revision control (i.e., where you could, for example, see diffs to old releases or who changed which test for what reason).

Keep in mind that tests are made up from modules so that you can eliminate duplication of test code; otherwise the maintenance will kill you. And before changing any module, you need to know where it is used.

In short: Make sure the master of your tests and your test data is in a text format that is versioned together with your code under test.

Nontechnical personnel (for example, management, QA) may require more high-level information about test coverage, latest test results, or even means of triggering a test run. Don’t let these valid requirements undermine the engineering approach to test automation. Instead, write little tools, for example, web-based report applications, that address these needs.

The ability of customers to access information about tests is as important as the ability to keep test and production code coordinated. As Dierk pointed out, you might not be able to do all this with the same tool.

Test management helps your team answer questions such as the following:

Which test cases have been automated?

Which still need automating?

Which tests are currently running as part of a regression suite?

Which tests cover what functional areas?

How is feature XYZ designed to work?

Who wrote this test case? When? Who changed it last?

How long has this test been part of the regression suite?

Because one of the primary reasons we write tests is to guide development, we need to organize tests so that everyone on the team can find the appropriate tests for each story and easily identify what functionality the tests cover. Because we use tests as documentation, it’s critical that anyone on either the development or customer team can find a particular test quickly when there’s a question about how the system should behave. We might need multiple tools to satisfy different test management goals.

It’s easy to lose control of test scripts. When a test fails, you need to pinpoint the problem quickly. You may need to know what changes have been made recently to the test script, which is easy with the history available in a source code control system. Your customer team also needs a way to keep track of project progress, to understand how much of the code is covered with tests, and possibly to run tests themselves. Test management systems, like the tests themselves, should promote communication and collaboration among team members and between different teams.

Test Transparency

Declan Whelan, a software developer and agile coach, uses a test management approach designed to keep tests visible to testers, developers, managers, and other teams.

We treat all test artifacts the same as source code from an organizational and revision control perspective. We use Subversion, and anyone who wants to run or edit the tests simply checks them out.

The latest Fit tests are available on a Confluence Wiki. We did this to support collaboration (team is distributed) and to leverage the strong capabilities of Confluence. Having the tests visible on the wiki was also helpful to others such as managers and other teams who did not want to check it out from the repository.

Prior to this, the QA team maintained test cases on a drive that was not accessible to anyone outside of QA. This meant that developers could not easily see what was being tested. Making the tests visible, transparent, and supported by a version control system (Subversion) really helped to break down barriers between developers and testers on the team.

Make sure your tests are managed with solid version control, but augment that with a way for everyone to use the tests in ways that drive the project forward and ensure the right value is delivered.