Crispin, Lisa - Agile Testing - A Practical Guide for Testers and Agile Teams

Testing product installations can also mean testing various installations of shrink-wrapped products to different operating systems or hardware. How does the product behave? Does it do what is expected? How long will the system need to be down for installation? Can we deploy without taking an outage? Can we make the user experience as pleasant as possible?

Janet’s Story

I had an experience a while ago that was not so pleasant, and it led me to wish that someone had tested and fixed the issue before I found it. I bought a new laptop and wanted to transfer my license for one of my applications to the new computer. It came with a trial version of the same application, so the transfer should have been easy, but the new PC did not recognize the product key—it kept saying it was invalid. I called the support desk and after a bit of diagnostics, I was informed they were considered different products, so the key wouldn’t work.

Two more hours of support time, and the issue was fixed. The trial version had to be removed, an old version had to be reinstalled, the key had to be reentered, and all updates since the original purchase had to be uploaded. How much easier would it have been for the development team to test that scenario and offer the customer an informative message saying, “The trial version is not compatible with your product key.” A message such as that would have let me figure out the problem and solve it myself rather than taking the support person’s time.

—Janet

Take the time you need to determine what your requirements are for testing installation. It will be worth it in the end if you satisfy your customers.

Communication

Constant communication between different development team members is always important, but it’s especially critical as we wrap up the release. Have extra stand-up meetings, if needed, to make sure everything is ready for the release. Write cards for release tasks if there’s any chance some step might be forgotten.

Lisa’s Story

My team releases after each iteration. We usually have a quick stand-up on the last afternoon of the sprint to touch base and identify any loose ends. Before the team had a lot of practice with releases, we wrote release task cards such as “run database update script in staging” and “verify database updates in production.” With more experience at deploying, we no longer need those cards unless we have a new team member who might need an extra reminder. It never hurts to have cards for release tasks, though.

—Lisa

Reminders of tasks, whether they are in a full implementation plan or just written on task cards as Lisa’s team does, are often necessary. On simple implementations, a whiteboard works well.

What If It’s Not Ready?

By constantly tracking progress in many forms, such as builds, regression test suites, story boards, and burndown charts, a team usually knows well in advance when it’s in trouble on a release. There’s time to drop stories and readjust. Still, last-minute disasters can happen. What if the build machine breaks on the last day of the iteration? What if the test database crashes so that final testing can’t be completed? What if a showstopper bug isn’t detected until final functional testing?

We strongly advise against adding extra days to an iteration, because it will eat into the next iteration or release development. An experienced team might be flexible enough to do this, but it can derail a new team. Still, desperate times call for desperate measures. If you release every two weeks, you may simply be able to skip doing the actual release, budget time into the next iteration to correct the problems and finish up, and release on the next scheduled date. If testing tasks are being put off or ignored and the release goes ahead, bring up this issue with the team. Did the testing needs change, or is the team taking a chance and sacrificing quality to meet a deadline? The team should cut the release scope if the delivery date is fixed and in jeopardy.

If your release cycle is longer, more like three months, you should know in advance if your release is in jeopardy. You probably have planned an end game of at least two weeks, which will just be for final validation. When you have a longer release cycle, you have more time to determine what you should do, whether it’s dropping functionality or changing the schedule.

If your organization requires certain functionality to be released on a fixed day and last-minute glitches threaten the release, evaluate your alternatives. See if you can continue on your same development cycle but delay the release itself for a day or a week. Maybe the offending piece of code can be backed out temporarily and a patch done later. The customers have the ultimate say in what will work for the business.

Lisa’s Story

On the rare occasions when our team has faced the problem of last-minute showstoppers, we’ve used different approaches according to the situation. If there’s nothing critical that has to be released right now, we sometimes skip the release and release two iterations’ worth on the next release day. If something critical has to go in, we delay the release a day or two. Sometimes we can go ahead and release what we have and do a patch release the next day. On one occasion, we decided to have a special one-week iteration to correct the problems, release, and then go back to the normal two-week iteration schedule.

After more than four years of practicing agile development, we have a stable build almost 100% of the time, and we feel confident about being able to release whenever it’s necessary. We needed a lot of discipline and continual improvement to our process in order to feel that a more flexible approach could work for us. It’s also nice to be able to release a valuable bit of functionality early, if we can. What we’ve worked hard to avoid is falling into a death spiral where we can never release on schedule and we’re always playing catch-up.

Don’t beat yourself up if you can’t release on time. Your team is doing its best. Do spend time analyzing why you got behind schedule, or over-committed, and take action to keep it from happening again.

—Lisa

Work to prevent a “no go” situation with good planning, close collaboration, driving coding with tests, and testing as you code. If your tracking shows the release could be in jeopardy, remove the functionality that can’t be finished, if possible. If something bad and unexpected happens, don’t panic. Involve the whole team and the customer team, and brainstorm about the best solution.

Customer Testing

There are a couple of different ways in which to involve your customers to get their approval or feedback. User Acceptance Testing can be fairly formal, with sign-offs from the business. It signifies acceptance of a release. Alpha or beta testing is a way to get feedback on a product you are looking to release but which is not quite ready.

UAT

User Acceptance Testing (UAT) is important in large customized applications as well as internal applications. It’s performed by all affected business departments to verify usability of the system and to confirm existing and new (emphasis on new) business functionality of the system. Your customers are the ones who have to live with the application, so they need to make sure it works on their system and with their data.

In previous chapters we’ve often talked about getting the customers involved early, but at those times, the testing is done on specific features under development. UAT is usually done after the team decides the quality is good enough to release. Sometimes though, the timeline dictates the release cycle. If that is the case, then try moving the UAT cycle up to run parallel with your end game. The application should be stable enough so that your team could deploy to the customer’s test system at the same time as they deploy to staging.