June 10, 2009
Automated Smoke Testing
A smoke test (in software) is a quick test done after a build has completed successfully, but before QA fully tests it.Email sign up
Microsoft claims that after code reviews, smoke testing is the most cost effective method for identifying and fixing defects in software.Because the sooner we find a defect, the cheaper it is to fix it, we use a continuous build process utilizing QA Wizard Pro to automatically perform a smoke test each time a build finishes. This lets developers know right away if something that they checked in recently caused a problem. The process to implement an automated smoke test will vary depending on your application and the configuration of your build tool, but the basic steps should remain the same.
- After the build has completed successfully, you may need to perform setup steps before testing your application. This may include copying files to the appropriate places, setting up database tables, installing licenses, or starting a server.
- A second setup step is to get all of the QA Wizard Pro files required for your smoke test. We do this using Surround SCM's command line tool to fetch several smoke test files to the local drive.
- To maintain the most flexibility in the smoke test while keeping the build script static, we have a single script that is responsible for performing each test. We can then start the smoke test from our build tool with a line similar to:
"C:Program FilesSeapineQA Wizard ProQAWRunScript.exe" "SmokeTestSmokeTest.qawwspace" "SmokeTestSmokeTest.qawscript" /Reportfile "BuildCurrentReport.qawreport"The smoke test is then run and its report is saved with the rest of the build files. Should something fail in the smoke test, QAWRunScript.exe returns an error code of 1 which is picked up by the build tool and then reported to the developers along with the output of the script.
- A final step is to clean up after the smoke test. This might include stopping a server, deleting files, or emptying database tables. This could also be done before the initial setup step to ensure that the environment is clean before any tests are started.