Crispin, Lisa - Agile Testing - A Practical Guide for Testers and Agile Teams

If you’re using a third-party product as part of your solution, you might assume it has been tested, but that might be a poor assumption. You will need to budget extra time to test your application in conjunction with the vendor software. If there’s a problem in the other company’s software, it might take a long time to resolve. Lisa’s team uses third-party software for critical tasks such as document creation. If a theme includes modifying or creating new documents, they plan extra time to upgrade the software if needed, and extra time for testing in case fixes are needed. If possible, bring the third-party software into the project early, and start end-to-end testing. The more you can work with the interface, the better off you’ll be.

Other third-party software that we often forget about until it’s too late is our own testing environments. Sometimes a team will incorporate new code that takes advantages of new features in their chosen language. For example, if team members are using AJAX or JavaScript, they may need to upgrade the software development kit they’re using. This means that a team will have to upgrade its production runtime environment as well, so take that into consideration and test early.

Clients or partners might have concerns about a release that isn’t within your own team’s scope. Lisa’s team was once prevented at the very last minute from releasing a new feature because a partner didn’t have time to okay the change with their legal advisors. The programmers had to quickly devise a way to turn the functionality off without requiring extensive additional testing. Interestingly, partners who aren’t using agile development sometimes have trouble meeting their own deadlines. They might be unprepared when your team meets the deadline.

Janet’s Story

I worked on a project to implement a feature that required a new piece of hardware for scanning a new 2D bar code. The team decided to implement in stages because it was not known when the scanners would be available for full testing, but the customer wanted the code ready for when the scanners arrived.

The initial phase was programmer-intensive because there was a lot of research to be done. After they determined how they would implement the feature, the story was created to add it into the code. However, we knew we couldn’t thoroughly test it until the scanners were available. The code was ready to test, but instead of backing it all out, we only needed to worry about testing that the feature could be turned off for the release. The next release would require more testing, but only if the scanners were available. The testing of the story was kept in the product backlog so we would not forget to do it.

—Janet

If you’ll be working with other teams developing different components of the same system, or related systems, budget time to coordinate with them. It’s a good idea to designate a member from each team to coordinate together.

Release planning is the time to identify extra roles you need on your team, additional resources, and time needed for out-of-the-ordinary circumstances.

Test Planning

We can’t expect to plan the iterations in a release at a detailed level. We can get an idea of the theme’s steel threads, prioritize stories, and make a guess at what stories will be in which iteration. Detailed test planning needs to wait for iteration planning. Still, we need to think about testing at a high level, and try to budget enough time for it. We might even take time separately from the release planning meeting to strategize our testing for the release. In Chapter 8, Business-Facing Tests that Support the Team, we mentioned one of the perils of agile testing: “forgetting the big picture.” Test Planning will help you with that problem.

Chapter 8, “Business-Facing Tests that Support the Team,” explains how to identify steel threads or thin slices in a story or theme.

Where to Start

During release planning, it’s helpful to know the business conditions of satisfaction for each story or high-level user acceptance test case. When stories need clarification, agile testers ask for examples. At this stage, examples will be high-level, covering just the basics, but enough to be able to size and prioritize the story. Drawing flowcharts or writing calculations on the whiteboard and discussing them helps us identify project-specific testing issues.

At a minimum, the team needs to understand the top-priority stories that are scheduled to be performed first. Lightweight planning might involve only looking at those core stories with the understanding that more time will be needed for defining additional tests.

As we get a sense of which stories will probably be included in the release, we can start thinking about the scope of the testing. What assumptions have been made that might affect testing? Use of third-party software, such as the example of using a shipping company’s shipping calculation API, affects test planning. Are there any unusual risks in this release that will impact testing? If we have stories to implement batch jobs, and we’ve never had any batch processing in the system before, there are probably new frameworks that impact testing. We need to budget time to learn them.

Why Write a Test Plan?

In release planning, we talk about the purpose of the release, what’s in scope, and what assumptions we’re making. We do some quick risk analysis and plan our test approach to address those risks. We consider automation and what we need for test environments and test data. We certainly want to identify milestones and deliverables. Hmmm, this is starting to sound like . . . a test plan!

If, like ourselves, you spent time working in a traditional waterfall environment, you might have wasted time writing bulky test plans that nobody read and nobody bothered to maintain. In agile development, we want our test plans to serve our needs. Your customer might require a test plan for each release for compliance reasons. Even if it’s not a required deliverable, it can be useful. Keep it concise and lightweight. It only has to serve your purposes during this release. Address the testing issues that are specific to this release or project. Include your risk analysis and identify assumptions. Outline the critical success factors that your customer has identified. Think about what people need to know related to testing, and remove anything extraneous.

See Chapter 5, “Transitioning Typical Processes,” for more about test plans and test strategies.

Even if you don’t create a formal test plan, be sure you have made note of all these different testing factors involved in the release. You’ll want to keep them in mind during every iteration planning session. The biggest benefit in test planning is the planning itself. It allows you to consider and address issues such as test data requirements, infrastructure, or even what test results are required. Test planning is a risk mitigation strategy. Let’s consider some of these issues.

Types of Testing

In Part III, we covered the four quadrants of testing and talked about all of the different types of testing you can do during your project. Release planning is a good time to consider these different needs. Do you need to plan to bring in a load test tool, or will there be the need to build some kind of specialty test harness?

It could be that your next release is just an extension of your last, and you will just carry on creating your examples, automating your story tests, and doing the rest of the testing as you’ve been doing. You are one of the lucky ones. For those of you who are starting a brand new project with no previous processes in place, now is the time to consider what testing you will need. We don’t mean you have to decide how to test each story, but look at the big picture and think about the quadrants. Will you need to plan for a special UAT, or will the iteration demos be enough? It is important to raise these issues early so the team can plan for them.