July 9, 2013

Does Your Process Allow for Experimentation?

Helix ALM
Requirements Management
Two years ago, I attended a Better Software conference where Kent Beck was one of the keynote speakers. Beck was the inventor of Extreme Programming and one of original signers of the Agile Manifesto. His talk was on the acceleration of software releases and how teams could cope with that acceleration. Beck noted that software releases were speeding up in recent years, and looked at what it would mean for testers once releases occurred every month, every week, or even every day. He concluded that if we weren’t sure of the quality of a particular feature, we would release it anyway, and fix it on the next release, whether it be the next week or next day. On June 20 of this year, I participated in Swiss Requirements Day, where I presented Why Can’t We Write Great Requirements? as a track session. The English keynote (sessions were presented in either English or German), given by computer science professor Jan Bosch at the Chalmers University of Technology, was entitled Optimal User Experience Through Innovation Experiment Systems. His proposition was that requirements and design knowledge often can’t determine the best user experience for our software, especially when it comes to small issues that can have a big impact on usability. Instead, innovation occurs with a series of small, fast experiments. These experiments involve simple design differences that are available to many users simultaneous, with the ability to objectively measure critical results from those differences. For example, on an e-commerce site I might make the checkout button either red or green, and measure either the resulting dollar amount of purchases, or the percent who abandoned the sale, for each alternative. Don’t laugh! In his studies, Bosch noted significant differences in purchasing based simply on different color schemes. These two talks enabled me to understand the limitations of requirements engineering, UI design, and testing. There comes a time where requirements, UI design, and feature set leaves the realm of analysis and fact and instead becomes opinion, informed or not. We simply don’t know the right way to present something to the user, so instead we test it with an experiment. Of course, this approach is most realistic with web applications, where we can reach many users quickly and simultaneously, and make changes on our servers, rather than on thousands of desktops. When I worked on commercial software projects, we attempted (with limited success) to do early prototypes of UI designs and test them with prospective users. But UI testing resulted in subjective analyses for a small group of users, rather than objective data across hundreds or thousands of users in a short period of time. And it was not possible to make and deploy changes nearly as quickly as it is today. Experimentation offers testers a unique opportunity to get more involved with software design decisions. Because testing practices already involve a high degree of experimental design skills, such experiments can be set up and done by testers, who can record results in application lifecycle management software such as Seapine TestTrack to share with all project stakeholders. The ability to test small but critical design decisions could represent a significant future for software testing.