Crispin, Lisa - Agile Testing - A Practical Guide for Testers and Agile Teams

Infrastructure

While you are doing your test planning, you need to consider your infrastructure. Infrastructure can mean your continuous integration setup, test environments, and test database. It can mean how you promote your builds to your test environments. It might mean your test lab, if you have one, or having a separate server to run all your automation tests. These are generally pieces of infrastructure that need some lead time to get in place. This is the time to make a plan.

Lisa’s Story

Some types of testing might require extra effort. My team had a tool to do performance testing and some scripts, but we lacked a production-style environment where we could control all of the variables that might affect performance. For example, the test database was shared by testers, programmers, and two build processes. Slower performance might simply mean someone was running database-intensive tests. We used our staging environment to get a baseline, but it was missing some of the components of production. We set a six-month goal to acquire hardware and software for a proper test environment and get it set up. We wrote a task card or two each iteration to establish the environment step by step.

—Lisa

Whatever your needs are, make sure you understand them and can plan for what you need. If you don’t have the right infrastructure, then you will waste time trying to get it together and cause a bottleneck in mid-iteration.

Test Environments

As we look at the types of features in the next release, we might see the need for a whole new test environment. Think about specialized test environments you may need as well. Will you need more tools? Do you need to expand your test lab so that you can test with different browsers and operating systems? This is the time to think about all testing considerations.

If you’re planning your first release, test environments are a key consideration. You might need a story or iteration just to set up the infrastructure you need. We’ve started more than one project where the only place we could test was the development environment. We found that doesn’t work very well, because the environment is never stable enough for effective testing.

Just as programmers have their own sandboxes to work and test in, it works well if each tester has that same availability and control. We recognize that not all applications lend themselves to this, but at the very least, you need to know what build you’re testing. You also need test data that others will not walk over with their tests. If you don’t have a testing sandbox that’s under your own control, take time to plan what you need to establish for your test environments. Brainstorm with your team about how you can obtain the necessary hardware and software. It might take time, so develop a Plan B for getting something done while waiting for the infrastructure you need.

If you’re working on a large system, you may have to queue up along with other teams to get time on a test or staging environment that includes all of the various pieces of software with which yours must work. This staging environment should mimic your production system as much as possible. If your organization doesn’t have someone responsible for creating environments, your team might need extra roles dedicated to obtaining the test environments you need. These roles might involve working with other teams as well. Release planning is the time to consider all of these testing infrastructure requirements.

Test Data

Release or theme planning is also a good time to think about what test data you might need during the project.

Using test data that closely resembles real data is generally a good practice. Plan for the data you need. We’ve had the opportunity in several organizations to use a copy of production data. Real data provides a good base for different scenarios for exploratory testing. Production data may need to be “scrubbed” before it’s used for testing in order to remove any sensitive information such as identification or bank account numbers. The data needs to be altered to hide the original values but remain valid so that it doesn’t violate database restrictions. Because it takes time for database experts to port production data to a test environment, be sure they’re included in your planning.

Janet’s Story

In one of the organizations I was working with, we used two different baseline test data schemes. For our individual test environments, we used Fit fixtures to load predefined data. We tried to make this data as close to production as possible, but we also seeded it with some very specific test data. Every time we checked out a new version of code, we were able to reload a base set of data. In this way, we also tested the database schema as well to see if anything had changed.

For our more stable test environment where we wanted data persisted, we used the data migration scripts that the programmers developed as they made database changes. These migration scripts were eventually used for the initial cut over from production and by then we were pretty certain they were correct.

—Janet

Enlist your customers’ support in obtaining meaningful test data. If you’re working on a story that involves sending a file to a third-party vendor, your business expert can find out what data the vendor expects in the file. Lisa’s team developed features to allow retirement plan brokers to offer their customers portfolios of mutual funds. They asked the product owner to provide samples of portfolios, including a name, description, and set of funds for each. This helped them test with realistic data.

Test data tends to get stale and out of date over time. Older data, even if it came from production, may no longer accurately reflect current production data. A “passing” test using data that’s no longer valid gives a misleading sense of confidence. Continually review your test data needs. Refresh data or create it using a new approach, as needed.

Test data requirements vary according to the type of testing. Regression tests can usually create their own data or run against a small representational set of data that can be refreshed to a known state quickly. Exploratory testing may need a complete replica of production type data.

Chapter 14, “An Agile Test Automation Strategy,” explores different approaches to obtaining test data

Test Results

Different teams have different requirements for test result reporting. Think about how you are going to report test results at this stage of the game so that you can do so effectively when the time comes to do the actual reporting. Your organization may have audit compliance requirements, or maybe your customer just wants to know how you tested. Understand your needs so that you can choose the approach that is right for your team.

There are many ways to report test results. There are vendor tools that will record both automated and manual results. Your team may find a way to persist the results from tools such as Fit, or you may just choose to keep a big visible manual chart.

The approach that a few teams have taken is to create home-grown test result applications. For example, a simple Ruby application written with Ruby on Rails for the database or a MySQL database with a PHP front end can make a very simple but easy-to-use test management system.

A tool such as this can be very simple or can include added complexity such as the capability to categorize your tests. The important thing is the test results. If your automated tests record their pass or fail result along with the error, you have some history to help determine fragility of the test.

Your team can configure your automated build process to provide test results from each build, by email, or a feedback utility or web interface that team members can view online. Results over time can be summarized in a variety of formats that make progress visible. One of Lisa’s teams produced a daily graph of tests written, run, and passing that was posted in the team’s work area. Another produced a daily calendar with the number of unit tests passing every day. Even simple visual results are effective.