Crispin, Lisa - Agile Testing - A Practical Guide for Testers and Agile Teams

Production-Like Data

The ability to test a system that is as much like production as possible is essential to most software development teams. However, running a suite of automated regression tests against a copy of a production database would probably run too slowly to be useful feedback. Besides, you couldn’t really depend on any data remaining stable as you bring over new copies to stay up-to-date. Generally, when you’re talking about functional or end-to-end testing, a clone of the production database is most useful for manual exploratory testing.

Stress, performance, and load testing, which are automation–intensive, need an environment that closely simulates production in order to provide results that can translate to actual operations. Usability, security, and reliability are other examples of testing that needs a production-like system, although they may not involve much automation.

There is always a trade-off; your production database might be huge, so it is expensive and slow, but it provides the most accurate test data available. If your organization can afford hardware and software to store multiple copies of production data for testing purposes, this is ideal. Small companies may have resource constraints that might limit the amount of data that can be stored in test and staging environments. In this case, you’ll need to decide how much test data you can support and plan how to copy enough relevant data to make the test representative of what’s used in “real life.” Or you may consider making the investment in hardware, which is getting less expensive every day, to support a real production style environment. Otherwise, your test results might be misleading. As we mentioned with the canonical data, you may need to scrub the data before using it.

Data Migration

Data migration needs to be tested against a real database. The database upgrade scripts need to be run against real data and against the last known release of the database schema.

Testing a Database Migration

Paul Rogers, an automation test architect, tells this story of testing an eye-opening database migration [2008]:

Just yesterday, I ran a Rails migration against my test database. The developers had written it, tested it, and checked it using their development databases. My test database was probably 20,000 times larger. The migration for them took seconds. For me, well, I stopped it after three hours, at probably 10% complete. The programmers needed to redo their migration strategy.

I doubt this would have shown up on an in-memory database, so for me, a real database in this instance was definitely the right choice. In fact, this is likely to feed into things we need to consider before releasing, such as how long does a deployment take, or how long does the database update take. We can then use this to estimate how much down time we will need for the actual upgrade.

This is another example of how we must strike a balance between tests that deliver quick feedback and tests that realistically reflect events that might occur in production.

Understand Your Needs

If you understand the purpose of your tests, you can better evaluate your needs. For example, if you don’t need to test stored procedures or SQL queries directly for speed, consider tools such as in-memory databases, which work just like real databases but greatly speed up your tests. When you need to simulate the actual production environment, make a copy of the entire production database, if necessary. Quick feedback is the goal, so balance testing realistic scenarios with finding defects as efficiently as possible.

Evaluating Automation Tools

The first step in choosing an automation tool is to make a list of everything the tool needs to do for you. Let’s consider how you can decide on your test tool requirements.

Identifying Requirements for Your Automation Tool

After deciding on the next automation challenge to tackle, think about your tool needs. What tools do you already have? If you need additional ones, you probably want something that integrates well with your existing testing and development infrastructure. Do you need a tool to easily integrate into the continuous build process? Will your existing hardware support the automation you need to do? Setting up a second build process to run functional tests may require additional machinery.

Who’s going to use the test tool you’re hoping to implement? Will non-programmers be writing test cases? Do your programmers want a tool they feel comfortable with as well? Do you have distributed team members who need to collaborate?

Who will be automating and maintaining the tests? The skills already on your team are important. How much time do you have to get a tool installed and learn how to use it? If your application is written in Java, a tool that uses Java for scripting may be the most appropriate. Do team members have experience with particular tools? Is there a separate test team with expertise in a certain tool? If you’re starting the transition to agile development and you already have a team of test automators, it may make sense to leverage their expertise and keep using the tools they know.

Your tool requirements are dependent on your development environment. If you’re testing a web application, and the tool you choose doesn’t support SSL or AJAX, you may have a problem. Not every test tool can test web services applications. Embedded system testing can need different tools again. The case study in Chapter 12, “Summary of Testing Quadrants,” shows one way to use Ruby to test an embedded application.

Of course, the type of testing you’re automating is key. Security testing probably needs highly specialized tools. There are many existing open source and vendor tools for performance, so the job of selecting one isn’t overwhelming. As you master one challenge, you’ll be better prepared for the next. It took Lisa’s team a couple of years to develop robust regression test suites at the unit, integration, and functional levels. Performance testing was their next area of pain. Lessons learned from the earlier automation efforts helped them do a better job of identifying requirements for a test tool, such as ease of reporting results, compatibility with existing frameworks, and scripting language.

Write a checklist that captures all your tool requirements. Some of them might conflict with or contradict each other—“The tool needs to be easy enough so that customers can specify tests” or “The tests should be easy to automate.” Write them down so you can find the right balance. Then start doing your research.

One Tool at a Time

You’re going to need different tools to serve different purposes. Implementing new tools and learning the best way to use them can get overwhelming pretty quickly. Try one tool at a time, addressing your greatest area of pain. Give it enough time for a fair trial and evaluate the results. If it’s working for you, master that tool before you go on to the next area of pain and the next tool. Multitasking might work for some situations, but new technology demands full attention.

When you’ve settled on a tool to address a particular need, take a step back and see what else you need. What’s the next automation challenge facing your team? Will the tool you just selected for another purpose work for that need, too, or do you need to start a new selection process?

If you’ve decided to look outside your own organization for tools, the first step is to find time to try some out. Start with some basic research: Internet searches, articles and other publications about tools, and mailing lists are good places to get ideas. Compile a list of tools to consider. If your team uses a wiki or online forum tool, post information about tools and start a discussion about pros and cons.