February 9, 2011

When Do You Ship?

Reporting
If you haven’t been intimately involved in an application development project, this question probably evokes some puzzlement. You ship your software when it’s done, right? Well, maybe. There are a number of considerations that you take into account when deciding when to ship an application—is the coding complete, is the quality there, and is the business ready? The developers are responsible for determining when coding is complete, while the business owners have to determine if the application is ready to increase revenue or reduce costs. I’m interested in focusing on the second aspect—the assessment of application quality. We can point to things like defect rate or requirements met as shipping criteria, but those are still ambiguous terms. Any application is going to have defects, and some requirements will be met better than others.

Assessing Application Quality

What you need is a quantifiable definition of the quality required to be ready to ship. The definition will vary depending on the application. For a safety-critical application, the definition will be much more exacting than it would be for a non-revenue internal enterprise application. Even so, many of the things you measure will be the same. You need to create, define, manage, and test requirements, and determine (often with some subjectivity) if those requirements have been met. The team and the user community have to jointly determine, ideally before the project begins, the relative importance of requirements, and how to objectively measure their completion. One way that might be done is by looking at defects. There are several ways of collecting and analyzing defect data. You can look at absolute number of defects, or you can rank them by priority. You can map them against requirements, looking at the most critical defects against the most important requirements. Another related approach is to look at test cases and test case coverage. Are your tests comprehensive? Are they executing consistently and successfully? If your test cases accurately reflect the requirements, this is a reasonable way to assess whether you’re done. However you go about determining when the application is ready to ship, you need data. At a minimum, you need to track your requirements, test cases, and defects. You can even get fancy, plotting defect find versus fix rates, or tracing requirements to test cases to defects in one comprehensive matrix. Then you compare the results to the criteria you set at the beginning of the project. That’s the only way you know you’re ready to ship.

QUEST Conference - Boston

I’m speaking in-depth on this topic at the Quality Engineered Software and Testing (QUEST) Conference being held in Boston on April 6-8 2011. My session, A Practical Approach to Determining Application Deployment Readiness, is on Friday morning, April 8. I’ll also be exhibiting Seapine’s quality-centric ALM solution at the conference. Stop by if you’d like to see our solutions, discuss better ways to know when to ship, or just get some free swag! If you are coming to the conference, you can save 20% if you use the following link and use Discount Code QB2011-Seapine. Code is valid for Attendee-Regular registrations on the Three-Day Conference & Exhibitor Expo package. 20% off QUEST Registration