Crispin, Lisa - Agile Testing - A Practical Guide for Testers and Agile Teams

Figure 11-1 Quadrant 4 tests

Nonfunctional requirements include configuration issues, security, performance, memory management, the “ilities” (e.g., reliability, interoperability, and scalability), recovery, and even data conversion. Not all projects are concerned about all of these issues, but it is a good idea to have a checklist to make sure the team thinks about them and asks the customer how important each one is.

Our customer should think about all of the quality attributes and factors that are important and make informed trade-offs. However, many customers focus on the business side of the application and don’t understand the criticality of many nonfunctional requirements in their role of helping to define the level of quality needed for the product. They might assume that the development team will just take care of issues such as performance, reliability, and security.

We believe that the development team has a responsibility to explain the consequences of not addressing these nonfunctional or cross-functional requirements. We’re really all part of one product team that wants to deliver good value, and these technology-oriented factors might expose make-or-break issues.

Many of these nonfunctional and cross-functional issues are deemed low-risk for many applications and so are not added to the test plan. However, when you are planning your project, you should think about the risks in each of these areas, address them in your test plan, and include the tools and resources needed for testing them in your project plan.

Lisa’s Story

In the past, I’ve been asked by specialists in areas such as performance and security testing why they didn’t hear much about “ility” testing at agile conferences or in publications about agile development. Like Janet, I’ve always seen these areas of testing as critical, so this wasn’t my perception. But as I thought about it, I had to agree that this wasn’t a much-discussed topic at the time (although that’s changed recently).

Why would agile discussions not include such important considerations as load testing? My theory is that it’s because agile development is driven by customers, from user stories. Customers simply assume that software will be designed to properly accommodate the potential load, at a reasonable rate of performance. It doesn’t always occur to them to verbalize those concerns. If not asked to address them, programmers may or may not think to prioritize them. I believe that one area where testers have contributed greatly to agile teams is in bringing up questions such as, “How many concurrent users should the application support?” and “What’s the average response time required?”

—Lisa

Because the types of testing in this quadrant are so diverse, we’ll give examples of tools that might be helpful as we go along instead of a separate toolkit section. Tools, whether homegrown or acquired, are essential to succeed with Quadrant 4 testing efforts. Still, the people doing the work count, so let’s consider who on an agile team can perform these tests.

Who Does It?

All of the agile literature talks about teams being generalists; anyone should be able to pick up a task and do it. We know that isn’t always practical, but the idea is to be able to share the knowledge so that people don’t become silos of information.

However, there are many tasks that need specialized knowledge. A good example is security testing. We’re not talking about security within an application, such as who has access rights to administer it. Because that type of security is really part of the functional requirements and will be covered by regular stories, verifying that it works falls within the first three quadrants. We’re talking about probing for external security flaws and knowing the types of vulnerabilities in systems that hackers exploit. That is a specialized skill set.

Performance testing can be done by testers and programmers collaborating and building simple tools for their specific needs. Some organizations purchase load-testing tools that require team members who specialize in that tool to build the scripts and analyze and interpret the results. It can be difficult for a software development organization, especially a small one, to have enough resources to duplicate an accurate production-level load for a test, so external providers of performance testing may be needed.

Larger organizations may have groups such as database experts that your team can use to help with data conversion, security groups that will help you identify risks to your application, or a production support team that can help you test recovery or failover. Build a close relationship with these specialists. You’ll need to work together as a virtual team to gather the information you need about your product.

Chapter 15, “Tester Activities in Release or Theme Planning,” explains how to plan to work with external teams.

The more diverse the skill sets are in your team, the less likely you are to need outside consultants to help you. Identify the resources you need for each project. Many teams find that a good technical tester or toolsmith can take on many of these tasks. If someone already on the team can learn whatever specialized knowledge is required, great; otherwise, bring in the expertise you need.

Skills within the Team

Jason Holzer, Product Owner for Property Testing (performance, security, stability, and reliability) at Ultimate Software, tells us that a good programmer can write a multithreaded engine to call a function concurrently and test performance. Jason feels that agile teams do have the skills to do their own performance testing; they just may not realize it.

Performance testing does require a controlled, dedicated environment. Some specialized tools are needed, such as a profiler to measure code performance. But, in Jason’s view, performance, stability, scalability, and reliability (PSR) tests can, and should, be done at the unit level. There’s a mind-set that holds that these tests are too complex and require specialists when in fact the teams do possess the necessary skills.

Jason finds that awareness of the “PSR” aspects of code needs to be part of the team’s culture.

If stakeholders place a high priority on performance, stability, scalability, and the like, Jason recommends that the team talk about ways to verify these aspects of the application. When teams understand the priority of qualities such as performance and reliability, they figure out how to improve their code to ensure them. They don’t need to depend on an outside, specialized team. Jason explains his viewpoint.

The potential resistance I see today to this plan is that someone believes that programmers don’t know how to PSR test and that there will need to be a great deal of training. In my opinion, a more accurate statement is that programmers are not aware that PSR testing is a high priority and a key to quality. I don’t think it has anything to do with knowing how to PSR test. PSR testing is a combination of math, science, analysis, programming, and problem solving. I am willing to bet that if you conducted a competition at any software development organization where you asked every team to implement a tree search algorithm, and the team with the fastest algorithm would win, that every team will do PSR testing and provide PSR metrics without teaching them anything new.

PSR testing is really just telling me “How fast?” (performance), “How long?” (stability), “How often?” (reliability), and “How much?” (scalability). So, as long as the awareness is there and the organization is seriously asking those questions with everything they develop, then PSR testing is successfully integrated into a team.