Crispin, Lisa - Agile Testing - A Practical Guide for Testers and Agile Teams

Take a second look at the skills that your team already possesses, and brainstorm about the types of “ility” testing that can be done with the resources you already have. If you need outside teams, plan for that in your release and iteration planning.

Regardless of whether or not your team brings in additional resources for these types of tests, your team is still responsible for making sure the minimum testing is done. The information these tests provide may result in new stories and tasks in areas such as changing the architecture for better scalability or implementing a system-wide security solution. Be sure to complete the feedback loop from tests that critique the product to tests that drive changes that will improve the nonfunctional aspects of the product.

Just because this is the fourth out of four agile testing quadrants doesn’t mean these tests come last. Your team needs to think about when to do performance, security, and “ility” tests so that you ensure your product delivers the right business value.

When Do You Do It?

As with functional testing, the sooner technology-facing tests that support the team are completed, the cheaper it is to fix any issues that are found. However, many of the cross-functional tests are expensive and hard to do in small chunks.

Technical stories can be written to address specific requirements, such as: “As user Abby, I need to retrieve report X in less than 20 seconds so that I can make a decision quickly.” This story is about performance and requires specialized tests to be written, and it can be done along with the story to code the report, or in a later iteration.

Consider a separate row on your story board for tasks needed by the product as a whole. Lisa’s team uses this area to put cards such as “Evaluate load test tools” or “Establish a performance test baseline.” Janet has successfully used different colored cards to show that the story is meant for one of the expert roles borrowed from other areas of the organization.

Some performance tests might need to wait until much of the application is built if you are trying to baseline full end-to-end workflows. If performance and reliability are a top priority, you need to find a way to test those early in the project. Prioritize stories so that a steel thread or thin slice is complete early. You should be able to create a performance test that can be run and continue to run as you add more and more functionality to the workflow. This may enable you to catch performance issues early and redesign the system architecture for improvements. For many applications, correct functionality is irrelevant without the necessary performance.

The time to think about your nonfunctional tests is during release or theme planning. Plan to start early, tackling small increments as needed. For each iteration, see what tasks your team needs in order to determine whether the code design is reliable, scalable, usable, and secure. In the next section, we’ll look at some different types of Quadrant 4 tests.

Performance Testing from the Start

Ken De Souza, a software developer/tester at NCR [2008], responded to a question on the agile-testing mailing list about when to do stress and performance testing in an agile project with an explanation of how he approaches performance testing.

I’d suggest designing your performance tests from the start. We build data from the first iteration, and we run a simple performance test to make sure it all holds together. This is more to see that the functionality of the performance scripts holds together.

I used JMeter because I can hook FTP, SOAP, HTTP, RegEx, and so on, all from a few threads, with just one instance running. I can test out my calls right from the start (or at least have the infrastructure in place to do it).

My eventual goal is that when the product is close to releasing, I don’t have to nurse the performance test; I just have to crank up the threads and let go. All my metrics and tasks have already been tested out for months, so I’m fairly certain that anyone can run my performance test.

Performance testing can be approached using agile principles to build the tools and test components incrementally. As with software features, focus on getting the performance information you need, one small chunk at a time.

“ility” Testing

If we could just focus on the desired behavior and functionality of the application, life would be so simple. Unfortunately, we have to be concerned with qualities such as security, maintainability, interoperability, compatibility, reliability, and installability. Let’s take a look at some of these “ilities.”

Security

OK, it doesn’t end in -ility, but we include it in the “ility” bucket because we use technology-facing tests to appraise the security aspects of the product. Security is a top priority for every organization these days. Every organization needs to ensure the confidentiality and integrity of their software. They want to verify concepts such as no repudiation, a guarantee that the message has been sent by the party that claims to have sent it and received by the party that claims to have received it. The application needs to perform the correct authentication, confirming each user’s identity, and authorization, in order to allow the user access only to the services they’re authorized to use. Testing so many different aspects of security isn’t easy.

In the rush to deliver functionality, both business experts and development teams in newly started organizations may not be thinking of security first. They just want to get some software working so they can do business. Authorization is often the only aspect of security testing that they consider as part of business functionality.

Lisa’s Story

My current team is a case in point. The business was interested in automating functionality to manage 401(k) plans. They did take pains to secure the software and data, but it wasn’t a testing priority. When I “got religion” after hearing some good presentations about security testing at conferences, I bought a book on security testing and started hacking around on the site. I found some serious issues, which we fixed, but we realized we needed a comprehensive approach to ensuring security. We wrote stories to implement this. We also started including a “security” task card with every story so that we’d be mindful of security needs while developing and testing.

—Lisa

Budgeting this type of work has to be a business priority. There’s a range of alternatives available, depending on your company’s priorities and resources. Understand your needs and the risks before you invest a lot of time and energy.

Janet’s Story

One team that I worked with has a separate corporate security team. Whenever functionality is added to the application that might expose a security flaw, the corporate team runs the application through a security test application and produces a report for the team. It performs static testing using a canned black-box probe on the code and has exposed a few weak areas that the developers were able to address. It does not give an overall picture of the security level for the application, but that was not deemed a major concern.

—Janet

Testers who are skilled in security testing can perform security risk-based testing, which is driven by analyzing the architectural risk, attack patterns, or abuse and misuse cases. When specialized skills are required, bring in what you need, but the team is still responsible for making sure the testing gets done.

There are a variety of automated tools to help with security verification. Static analysis tools, which can examine the code without executing the application, can detect potential security flaws in the code that might not otherwise show up for years. Dynamic analysis tools, which run in real time, can test for vulnerabilities such as SQL injection and cross-site scripting. Manual exploratory testing by a knowledgeable security tester is indispensable to detect issues that automated tests can miss.