February 27, 2013

Test Automation is No Substitute for the Skilled Tester

Test Management
There has been a lot of angst in the testing blogosphere lately on the value of the skills and results that experienced testers bring to a project. For examples, see Pete Walen, Bruce Rennie (this is a very long thread, and may require LinkedIn login), and Keith Klain. All rail against a seemingly growing sentiment within software organizations that testing—especially on-site, experienced, manual testing—adds little value to the task of delivering software to the business. Most testers who write about this topic painstakingly attempt to define how testers explore software to determine how it works and where the biggest risks lay; and work with the user community to understand how they work and which actual or potential features are most important to them in their work. Testing legend James Bach has been one of the longtime proponents of the thinking tester, and the thinking tester was the theme of last year’s CAST Conference. First, let’s be clear: No one is trying to throw out automation. Automation takes dull and repetitive tasks out of the hands of the tester, leaving the individual with more opportunity to think, investigate, and provide recommendations. That’s largely a good thing, as it lets software do the things it does best, and the tester what he or she does best. The question is one of replacing the tester with automation. If all that your testers are doing is manually running through prepared test scripts, replacing them with automated testing might make sense. But if that’s all your testers are doing, then you aren’t really doing testing. By that I mean you don’t have people who can analyze and explain risks, and make recommendations on the readiness of software. As Marcus Gartner puts it, we need human testing for our second order of ignorance, to investigate the things that we don’t know we don’t know. Automation helps us find common problems and find regressions, which is an important part of the testing process, but human investigation and insight is essential in looking beyond the surface. I’ve spoken of the need for automation in my Moneyball-themed articles and presentations. Daniel Kahneman explains that his System 1 thinking is reflexive and automatic. System 1 activities like repeated functional and regression testing from scripts are the types of activities that can successfully be automated, and automation standardizes these activities to reduce bias. But he also defines a System 2 type of thinking that requires we engage conscious thought and actually work through a problem to make a decision. Skilled, qualified testers can work beyond sterile automation results to make sense of a project as a whole, and make recommendations on that project. Unless organizations believe that the metrics tell them all they need to know, they need skilled and experienced testers to discover just what it is they don’t know. If you’re a skilled tester, Seapine’s Defect Scribe can help you leverage those skills with exploratory testing while also capturing the steps needed to reproduce a defect.