Crispin, Lisa - Agile Testing - A Practical Guide for Testers and Agile Teams

Monitoring Tools

Tools like the Unix/Linux command tail -f, or James Bach’s LogWatch, can help monitor log files for error conditions. IDEs also provide log analysis tools. Many error messages are never displayed on the screen, so if you’re testing via the GUI, you never see them. Get familiar with tools like these, because they can make your testing more effective and efficient. If you are not sure where your system logs warnings and errors, ask your developers. They probably have lots of ideas about how you can monitor the system.

Simulators

Simulators are tools used to create data that represent key characteristics and behavior of real data for the system under test. If you do not have access to real data for your system, simulated data will sometimes work almost as well. The other advantage of using a simulator is for pumping data into a system over time. It can be used to help generate error conditions that are difficult to create under normal circumstances and can reduce time in boundary testing.

See the “System Test” example in Chapter 12, “Summary of Testing Quadrants,” to see how a simulator was critical to the testing the whole system.

Setting up data and test scenarios is half of the picture. You also need to have a way to watch the outcomes of your testing. Let’s consider some tools for that purpose.

Emulators

An emulator duplicates the functionality of a system so that it behaves like the system under test. There are many reasons to use an emulator. When you need to test code that interfaces with other systems or devices, emulators are invaluable.

Two Examples of Emulators

WestJet, a Canadian airline company, provides the capability for guests to use their mobile devices to check in at airports that support the feature. When testing this application, it is better for both the programmers and the testers to test various devices as early as possible. To make this feasible, they use downloadable emulators to test the Web Check-in application quickly and often during an iteration. Real devices, which are expensive to use, can then be used sparingly to verify already tested functionality.

The team also created another type of emulator to help test against the legacy system being interfaced with. The programmers on the legacy system have different priorities and delivery schedules, and a backlog of requests. To prevent this from holding up new development, the programmers on the web application have created a type of emulator for the API into the legacy system that returns predetermined values for specific API calls. They develop against this emulator, and when the real changes are available, they test and make any modifications then. This change in process has enabled them to move ahead much more quickly than was previously possible. It has proved to be a simple but very powerful tool.

Emulators are one tool that helps to keep testing and coding moving together hand-in-hand. Using them is one way for testing to keep up with development in short iterations. As you plan your releases and iterations, think about the types of tools that might help with creating production-like test scenarios. See if you can use the tools you’re already using for automating tests to drive development as aids to exploratory testing.

Driving development with tests is critical to any project’s success. However, we humans won’t always get all of the requirements for desired system behavior entirely correct. Our business experts themselves can miss important aspects of functionality or interaction with other parts of the system when they provide examples of how a feature should work. We have to use techniques to help both the customer and developer teams learn more about the system so they can keep improving the product.

Summary

A large part of the testing effort is spent critiquing the product from a business perspective. This chapter gave you some ideas about the types of tests you can do to make your testing efforts more effective.

Demonstrate software to stakeholders in order to get early feedback that will help direct building the right stuff.

Use scenarios and workflows to test the whole system from end to end.

Use exploratory testing to supplement automation and to take advantage of human intellect and perceptions.

Without usability in mind when testing and coding, applications can become shelfware. Always be aware of how the system is being used.

Testing behind the GUI is the most effective way of getting at the application functionality. Do some research to see how you can approach your application.

Incorporate all kinds of tests to make a good regression suite.

Don’t forget about testing documentation and reports.

Automation tools can perform tedious and repetitive tasks, such as data and test scenario setup, and free up more time for important manual exploratory testing.

Tools you’re already using to automate functional tests might also be useful to leverage exploratory tests.

Monitoring, resource usage, and log analysis tools built into operating systems and IDEs help testers appraise the application’s behavior.

Simulators and emulators enable exploratory testing even when you can’t duplicate the exact production environment.

Even when tests are used to drive development, requirements for desired behavior or interaction with other systems can be missed or misunderstood. Quadrant 3 activities help teams keep adding value to the product.

Chapter 11 Critiquing the Product Using Technology-Facing Tests

This chapter is focused on the bottom right corner of our testing quadrant. We’ve looked at driving development with both business-facing and technology-facing tests. After the code is written, we are no longer driving the development but are looking at ways to critique the product. In the previous chapter, we examined ways to critique from a business point of view. Now we look at ways to critique from a technology-facing point of view. These tests are an important means of evaluating whether our product delivers the right business value.

Introduction to Quadrant 4

Individual stories are pieces of the puzzle, but there’s more to an application than that. The technology-facing tests that critique the product are more concerned with the nonfunctional requirements than the functional ones. We worry about deficiencies in the product from a technical point of view. Rather than using the business domain language, we describe requirements using a programming domain vocabulary. This is the province of Quadrant 4 (see Figure 11-1).