Crispin, Lisa - Agile Testing - A Practical Guide for Testers and Agile Teams

Chapter 18, “Coding and Testing,” explores metrics related to defect rates.

Traceability

Another reason we’ve heard for having a DTS is traceability, linking defects to test cases. We’re not sure that this is a valid reason. Not all defects are linked to test cases, nor should they be. For example, errors like spelling mistakes might not need specific test cases. Maybe the product was not intuitive to use; this is a very real bug that often goes unreported. How do you write a test to determine if something is usable? Exploratory testing might find bugs in edge conditions that are not worth the effort of creating automated tests.

If it is an automated test case that caught a bug, then the need to record that defect is further reduced, because it will be caught again if ever reintroduced. The need for traceability is gone. So, maybe we don’t need to track defects.

Why Shouldn’t We Use a DTS?

Agile and Lean provide us with practices and principles that help reduce the need for a DTS. If the process is solid, and all of the people are committed to delivering a quality product, defects should be rare and very simply tracked.

As a Communication Tool

Defect tracking systems certainly don’t promote communication between programmers and testers. They can make it easy to avoid talking directly to each other.

Waste of Time and Inventory

We tend to put lots of information into the DTS in addition to all of the steps to reproduce the defect. Depending on the bug, it can take a long time to write these steps so that the programmer can reproduce it as well. Then there is the triage, and someone has to make comments, interpret the defect, attempt to reproduce it, (ideally) fix it, write more comments, and assign it back to the person who reported it. Finally, the fix can be verified. This whole cycle can double if the programmer misunderstood the problem in the first place. The cost of a single defect report can become exorbitant.

How much easier would it be if we as testers could just talk to the programmer and show what we found, with the developer then fixing the defect right away? We’ll talk more about that later.

In Chapter 15, “Coding and Testing,” we’ll explain how tester and programmers work together on bugs.

Defects in a DTS become a queue or a mini product backlog. According to lean principles, this inventory of defects is a waste. As a team, we should be thinking of ways to reduce this waste.

Janet’s Story

In 2004, Antony Marcano, author of TestingReflections.com, wrote a blog post about the idea of not using a bug-tracking system. When it was discussed on mailing lists, he was flamed by many testers as introducing something similar to heresy. He finds he has a different reception now, because the idea is making its way into the mainstream of agile thinking.

He suggests that bug-tracking systems in agile teams are just “secret backlogs.”

—Janet

Antony will share his ideas about the hidden backlog when we cover iteration planning in Chapter 18, “Coding and Testing.”

Defect Tracking Tools

If you do decide to use a DTS, choose it carefully. Understand your needs and keep it simple. You will want everyone on the team to use it. If it becomes overhead or hard to use, people will find ways to work around it. As with all tools used by your agile development team, you should consider the whole team’s opinion. If anyone from the customer team enters bug reports, get his or her opinion too.

One of the simplest tools that Janet has used is Alcea’s FIT IssueTrack. It is configurable, does not make you follow a predefined process, and is easy to get metrics out of. Do your homework and find the tool that works for you. There are a variety of open source defect-tracking systems, hosted systems, and integrated enterprise systems available.

Whether or not you use a DTS, you want to make defects as visible as possible.

We use a commercial DTS, but we find value in keeping bugs visible. We color-code bugs and include them as tasks in our story board, shown in Figure 5-1. Yellow cards denote normal bugs, and red cards denote either high production bugs or “test stopper” development bugs—both categories need to be addressed right away. A quick look at the board lets us see how many bugs are in the To Do, WIP, Verify and Done columns. Other cards are color-coded as well: blue for story cards, green for test task cards, and white for development tasks. Striped cards are for tasks added after iteration planning. Yellow and red bug cards stand out easily.

Figure 5-1 Story board with color-coded cards. Used with permission of Mike Thomas. Copyright 2008.

During the time we were writing this book, my team converted to a virtual story board because one of our team members became a remote team member, but we retained this color-coding concept.

—Lisa

We usually recommend experimenting with different tools, using each one for a few iterations, but this is trickier with bug-tracking systems, because you need to port all of the bugs that are in one system to the new one that you’re trying on for size. Spend some time thinking about what you need in a DTS, what purposes it will serve, and evaluate alternatives judiciously.

Lisa’s Story

My team used a web-based DTS that was basically imposed upon it by management. We found it somewhat cumbersome to use, lacking in basic features such as time-stamping updates to the bug reports, and we chafed at the license restrictions. We testers were especially frustrated by the fact that our license limited us to three concurrent users, so sessions were set to time out quickly.

The team set aside time to evaluate different DTS alternatives. At first, the selection seemed mind-boggling. However, we couldn’t find one tool that met all our requirements. Every tool seemed to be missing something important, or we heard negative reports from people who had used the tool. We were concerned about the effort needed to convert the existing bug database into a new system.

The issue was forced when our DTS actually crashed. We had stopped paying for support a couple of years earlier, but the system administrator decided to see what enhancements the vendor had made in the tool. He found that a lot of shortcomings we had experienced had been addressed. For example, all updates were now time stamped. A client application was available that wasn’t subject to session timeouts and had enhanced features that were particularly valuable to the testers.

By going with our existing tool and paying for the upgrade and maintenance, plus a license allowing more concurrent users, we got help with converting our existing data to the new version and got a working system easily and at a low cost. A bonus was that our customers weren’t faced with having to learn a new system.

Sometimes the best tool is the one you already have if you just look to see how it has improved!

—Lisa

As with all your tool searches, look to others in your community, such as user groups and mailing lists, for recommendations. Define your criteria before you start looking, and experiment as much as you can. If you choose the wrong tool, cut your losses and start researching alternatives.

Keep Your Focus

Decisions about reporting and tracking defects are important, but don’t lose track of your main target. You want to deliver the best quality product you can, and you want to deliver value to the business in a timely manner. Projects succeed when people are allowed to do their best work. Concentrate on improving communication and building collaboration. If you encounter a lot of defects, investigate the source of the problem. If you need a DTS to do that, use it. If your team works better by documenting defects in executable tests and fixing them right away, do that. If some combination enables you continually improve, go with it. The main thing to remember is that it has to work for your whole team.