August 15, 2014

Don't Have a Human Do a Machine's Work

Test Management
[caption id="attachment_14784" align="alignright" width="250"](photo credit: Paramount Pictures) Photo credit: Paramount Pictures[/caption] Recently, we published “Reducing Risk with Exploratory Testing,” a white paper that discusses a few ways adding exploratory testing to your testing regimen can improve your test coverage and help reduce risk. After we released the paper, I had an interesting conversation with Jeff Knee, who is Senior Software Test Engineer at Seapine. He pointed out something we don’t directly discuss in the paper, which is that testers running scripted tests can put a lot of effort into performing the steps without actually addressing the intent of the test. “It’s the forest of ‘verify that X can do Y,’” Jeff said, “versus the trees of ‘do step A, step B, and step C, before looking at D, and then continue with E.’”

The Testing Spectrum

Think of testing as a spectrum, with exploratory testing at one end and scripted testing at the other. The closer your tests are to the exploratory end, the more human involvement you need. The closer to the scripted end, the more you can replace a human tester with computer-run automated tests. This isn’t to say one is better than the other. Automated scripted tests are great for reducing the time it takes to run tests, increasing test coverage, and more effectively using human testers where they are needed most. Exploratory testing, on the other hand, often finds the most important bugs in the shortest amount of time. It allows testers to find more defects by giving them more latitude to try different types of tests, using their past experience and knowledge of the product. It frees a tester to explore the intent behind running the test, rather than just robotically following the steps.

Change Your Style

Exploratory tests have to be written less precisely than scripted tests, or they run the danger of becoming scripted tests. It’s the difference between “Verify that the search function works correctly,” versus the more scripted “Enter search term X. Verify that it returns result Y.” “One of the benefits of writing tests are that more exploratory in nature and less scripted,” Jeff said, “is that it can be much easier for the tester to understand the intent of the test case or test run.” You want your exploratory testers to be able to use that understanding to push the application to its limits and find the hidden defects a scripted test won’t find. Highly scripted test cases/runs, on the other hand, can lock the tester into a “do the steps only” mindset. If you go too far, your human tester might as well be an automaton.