Crispin, Lisa - Agile Testing - A Practical Guide for Testers and Agile Teams

Lisa’s Story

My current team made a commitment to automating regression tests at all levels where it made sense. Here are some examples of where I’ve asked for help to succeed with automation.

Early on, when we had no automated tests at all and the developers were trying to master test-driven development, we settled on Canoo WebTest for the GUI smoke tests. I needed a bit of help understanding how to configure WebTest to run in our environment, and I needed a lot of help to run the tests from the automated build process. I asked our system administrator (who was also one of the programmers) to help. We quickly got a suite of tests running in the build.

Later, I really wanted to try FitNesse for functional testing behind the GUI. I had to be patient while the programmers were still getting traction with the automated unit tests. The team agreed to try the tool, but it was hard to find time to start using it. I picked a story that seemed suited to FitNesse tests, and asked the programmer working on the story if I could pair with him to try some FitNesse tests. He agreed, and we got some tests automated in FitNesse. The programmer found it easy and worthwhile, and gave a good report to the rest of the team.

After that, it wasn’t hard to approach each programmer, suggest writing FitNesse tests for the story he was working on, and let him see the results. The FitNesse tests found test cases the programmer hadn’t thought of, and they saw the benefit right away. When everyone on the team had some experience with the tool, they were not only happy to automate the tests, but started designing code in a way that would make writing FitNesse fixtures easier.

When our Ruby expert, who designed most of our Watir test suite, left the company, I was quite concerned about maintaining our huge suite of tests as well as being able to code new ones. My Ruby expertise was not as good as his (plus, we were down to just one tester, so time was an issue). Every programmer on the team went out, bought a book on Ruby, and helped when I had problems updating scripts to work when the code changed. One programmer even wrote a new script to test a new story when I didn’t have time for that task. When we hired a new tester, he and I were able to handle the care and feeding of the Watir scripts, so the programmers no longer needed to take on those tasks.

I know I can ask teammates for help with automation issues, and the entire team sees automation as a priority, so the programmers always think about testability when designing the code. This is an example of the whole-team approach at work.

—Lisa

Specialized technology-facing tests such as security or load testing might require bringing in experts from outside the team. Some companies have specialist teams that are available as shared resources to product teams. Even while taking advantage of these resources, agile teams should still take responsibility for making sure all types of testing are done. They may also be surprised to find that team members may have the skills needed if they take a creative approach.

Chapter 11, “Critiquing the Product Using Technology-Facing Tests,” talks about technology-facing tests such as these and different approaches to handling them.

Some organizations have independent test teams that do post-development testing. They may be testing to ensure the software integrates with other systems, or conducting other specialized testing such as large-scale performance testing. Development teams should work closely with these other teams, using feedback from all testing efforts to improve code design and facilitate automation.

Taking the Time to Do It Right

Solving problems and implementing good solutions takes time. We must help our management understand that without enough time to do things the right way, our technical debt will grow, and our velocity will slow. Implementing solutions the “right” way takes time up front but will save time in the long term. Consider the time it takes for brainstorming ideas, solutions, formal training, and for on-the-job learning.

Your organization’s management is understandably interested in producing results as quickly as possible. If management is reluctant to give the team time to implement automation, explain the trade-offs clearly. Delivering some features in the short term without automated regression tests to make sure they keep working will have a big cost down the line. As your team accumulates technical debt, you’ll be less able to deliver the business value management needs. Work toward a compromise. For example, cut the scope of a feature but keep the essential value, and use automation to deliver and maintain a better product.

We always have deadlines, and we always feel pressed for time. The temptation to just go back to doing things the way we always have, like executing regression tests manually and hoping for the best, is always there even though we know that doesn’t work. There is never enough time to go back and fix things. During your next planning meeting, budget time to make meaningful progress on your automation efforts.

Lisa’s Story

Our team focuses on taking time for good design, a strong set of automated tests, and ample time for exploratory testing. Quality, not speed, has always been our goal. Our production problems cost a lot to fix, so the whole company is on board to take the time to prevent them. Sometimes we don’t pick the right design, and we aren’t afraid to rip it out and replace it when we realize it.

Naturally there are business tradeoffs, and the business decides whether to proceed with known risks. We work to explain all of the risks clearly and give examples of potential scenarios.

Here are a couple of recent examples of taking the time to do things right. We started a theme to make major changes to the account statements for the retirement plans. One of the programmers, Vince Palumbo, took on the task of collecting additional data to be used for the statements. He decided to write robust unit tests for the data collection functionality, even though this meant the story would have to continue on to the next iteration. Writing the unit tests took a great deal of time and effort, and even with the tests, the code was extremely complex and difficult to do. A couple of iterations later, another programmer, Nanda Lankalapalli, picked up another story related to the data collection and was pleasantly surprised to find new unit tests. He was able to make his changes quickly, and the testing effort was greatly reduced because the unit tests were in place.

Later, we found we had missed an edge case where some calculations for the change in account value were incorrect. The combination of automated unit tests and a great deal of exploratory testing were not enough to catch all of the scenarios. Still, having the tests meant Vince could write his corrected code test-first and feel more confident that the code was now correct.

Another recent example concerned processing of incoming checks. The business wanted to shorten the two-step process to one step, which meant the money would be invested in the retirement plan accounts two days earlier than was then possible. The existing process was all written in legacy code, without unit tests. We discussed whether to rewrite the processing in the new architecture. Our product owner was concerned about the amount of time this might take. We felt it would take just as long to change the existing code as to completely rewrite it, because the old code was difficult to understand and had no unit tests at all. We decided on the rewrite, which not only reduced the risk of problems in this critical functionality but also gave us the opportunity to provide a couple of extra features at little extra cost. So far, this strategy has proven worthwhile.