January 27, 2011

Test Earlier, Even With Incomplete Code

Test Management
In most companies, there are finite software release cycles. There is usually a deadline for a release to make it out the door. Schedules are often driven by business needs, whether it be to produce more revenue or to take advantage of a particular event, such as a holiday. Given a development cycle with a hard stop, it's up to the team and business owners to decide which features to include in the release and to ensure they are code complete and tested. It’s not unusual for development to take longer than anticipated, and this can cut into the QA time. This is especially true if QA does not get the first 'testable' build until all features are code complete. However, if developers can provide QA with feature complete (or almost complete) releases earlier in the process, they will be able to start testing those features earlier in the process. The testers have typically completed their test planning and test case development by this time, so they are ready to begin work. Thanks to the ability to get incomplete but functional builds earlier in the process, QA is able to test earlier in the process and, thus, test more. In many QA organizations, it is unrealistic to run every possible test on software because of time constraints. Testers focus on the most critical tests and, if time allows, they may run some of the non-critical tests as well. I like to use examples to better illustrate my arguments, so here is one: WysiCorp is working on the 2011 release of its WysiCRM application, which is scheduled to ship in time to provide upgrade revenue for the current business year. This new version will include six new features. Under the old model, they would code complete all six features, create an alpha build and give it to QA for testing. If the development schedule slipped at all, QA would find their backs against the wall, with too little time to do a credible job at assuring application quality. In the new model, the Dev team focuses on code completing one feature at a time. As soon as one feature is code complete, a build is created and handed to QA. So, instead of waiting for six features to be code complete, QA only has to wait for one. This allows them to start finding bugs on these features earlier in the process and gives development a chance to address bugs before the release. The development time does not change that much, and the six completed features are delivered in a similar timeframe. The big difference is that QA tests features as they are code complete and can fully focus on one feature at a time. QA is also able to run more in-depth tests, which allows them to test areas that might have gone untested. This is also a reason why it’s important to build early and often. The earlier builds are available, the more testing can occur. Keep in mind that the number of bugs found will increase with this approach. This should not be alarming. The more you test and the more areas you test, the more bugs you are going to find. This does not mean that your quality is suffering, it means you are catching more bugs before they make it out the door. And who would you rather find your bugs—your QA team or your customer? Another thing to keep in mind is that certain tests may be blocked from completing because the code isn’t complete. This is also to be expected, and it requires good test case management to make sure that blocked cases are tracked until they are successfully executed.