July 21, 2009

Which Tests Do I Automate?

Helix ALM
Test Management
One decision you may find yourself having to make when beginning automated testing is selecting which of your tests to automate. Automating all of your tests may sound appealing at first, but is both difficult and impractical. Some tests are better suited for manual testing while some make excellent candidates for automated testing. Here are some guidelines when looking at an individual test to determine if it's a good candidate for automation:
  1. How often is the test run, and would you benefit from having it run more often? A common test, like making sure you can login, is simple and often easy to automate. It also would be beneficial to find out right away if you cannot login, and could be run after every single build (as in a smoke test). A test that only needs to be run once before release isn't likely to be worth automating.
  2. How much data needs to be entered for the test (or how tedious is the test)? Adding one thousand entries to your database is a pain to do manually, but no sweat for an automated test. In general, the more data that's entered, the better the automation candidate.
  3. Is the output of the test easily measured? If you need to cross-reference several items to see if a test succeeded or failed, it will be more difficult to automate the test than if the output is easily visible in a screen after running the test.
  4. Is the output of the test purely objective? If the tester is required to make sure a result is exact, an automated test can usually do that. If, for example, the tester is required to make sure an image looks "good" or even "readable", an automated test will have a very difficult time verifying the image.
  5. Does the UI around the area being tested change a lot from version to version? An automated testing tool can handle some changes to controls, but likely could not handle the moving of a field to a completely different dialog without the automated script being modified to compensate for it.
  6. Does the test utilize any customized controls? If a test uses ordinary buttons, edit boxes, combo boxes, or grids, it stands a better chance of working as an automated test than a custom-built control does. Not that custom controls cannot be automated, but you may get better results automating with the more standard parts first.
  7. Does the test invite exploration of corner cases or improvisation? An automated test can only do what it is told; it is not creative and won't explore corner cases.
Note that the above are just guidelines, and even if your test meets all the guidelines for a good candidate for automation, it may still be difficult to automate the particular test. Are there any other guidelines that you have found when looking at a test to automate? Feel free to post them in the comment section below.