November 17, 2008
Analyzing the Script Agent
If you use another automated testing or scripting tool, you can also schedule and run automated script files through the use of a separate executable, known as a script agent, and a text file for communication between the script agent and TestTrack TCM. The script agent launches the script and looks for the exit code of the script. Based on the exit code of the script, the script agent returns a status to the text file. The text file, known as a *trjob file, communicates the status and results of the script to TestTrack TCM. We are going to disect the script agents available for download here, to help you use it with your automated testing scripts. This will be not be a line by line description of the source code of each script agent, rather, it will be a general walk through of both agents. Note: If your script can only be launched through a graphical user interface, you cannot use the script agent. You must be able to launch the script from the command line or programmatically. You can find out how to configure the script agent to launch your scripts here. [toc]
The ParameterWhen TestTrack TCM starts a script it calls the script agent. TestTrack TCM passes the script agent a single parameter, the full path to the script file.
The script file pathUnder the General category in Tools > Local Options, the TestTrack TCM Automated Scripts Directory is set. This directory is the working area for TestTrack to launch the script file and record the result in the .trjob file. Under this directory, TestTrack creates a directory named ScheduledScripts, and under it, TestTrack creates a directory with the name of the TestTrack project. If you have multiple projects with automated scripts, then you will have multiple directories. Under the project folder, a folder is created for each script in each test run. The naming of the folder is TR<test run number>_<script ID>_<Script name>. For example, if I run the "LoginTest.vbs" script that has a script ID of 1, and I run it for test run 28, the folder name is "TR28_1_LoginTest". TestTrack creates a copy of the script file and places it in this directory. The full path to the script passed to the script agent points to this file. Assuming that the TestTrack Automated Script Directory is set to C:scripts and that we are working with the sample project, the full path in this example is C:scriptsScheduledScriptsSample ProjectTR28_1_LoginTestLoginTest.vbs.
Trjob file pathTestTrack TCM looks for the trjob file in the same directory as the script. It assumes the trjob file has the same name as the script. In the example above, TestTrack TCM looks for a file named LoginExample.trjob in the C:scriptsScheduledScriptsSample ProjectTR28_1_LoginTest directory. The script agent takes the parameter (full path to the script file) and looks for the location of the period (the period preceding the file extension). It then takes the text up to the extension and appends "trjob" to the file. Note: The current code looks for the first instance of a period. If any of the directories in the script file path has a period, the script agent most likely will not work.
Execution commandThe QA Wizard Pro batch files are stand alone, which is why the script agent simply sends the full path the script file to the "system". If your script file is not stand alone (you must also tell the system which application is needed to run it), you can edit the command to also include the executable. The command would probably be something like "<Full path to executable> <Full path to script file>". The execution command is different between the QAWProBatAgent and the QAWBatAgentApp.
The results fileBoth have the "system" run the script, but the QAWProBatAgent also passes a parameter that indicates where to store the report file. The QAWBatAgentApp doesn't do this because the report is always created in a subdirectory under the location of the script file. The report is the results file. In the trjob file, the relative path to the report is set for the MainResultsFile. This allows you to view the report from TestTrack. You may have to edit this section of the code to accomodate the way your automated testing application handles reporting. Note: Neither the script agent or TestTrack creates a report. If your automated test tool does not create a report, you can't figure out how to programatically figure the path to the report for the script, or the report cannot be opened outside of the tool, you may want to simply comment out the section where the path to the results file is set.
Exit CodesWhen the script agent passes the execution command to the system, it captures the exit code to an integer. The script agent uses this integer to determine the status to enter in the trjob file. If the exit code is 0, then the status of the script in the trjob file is set to Passed. If the exit code is 1, then the status is set to Failed. For any other exit code, the status is set to Indeterminate.
ExampleYou most likely will not want to change the source code and use the script agent as is. Note that TestTrack installs exectuables for each version of the script agent. In the installation directory, you will see a ScriptAgents directory containing several agents, including QAWAppAgent.exe and QAWProAppAgent.exe. For this example, we will look at AutomatedQA's TestComplete. This tool allows you to create a batch file that you can use to run the script "stand alone". Following is an example batch file from their documentation:
REM Clears the screen CLS @ECHO OFF REM Launches TestComplete, REM executes the specified project REM and closes TestComplete when the run is over "C:Program FilesAutomated QATestComplete 6BinTestComplete.exe" "C:Program FilesAutomated QATestComplete 6SamplesScriptsHelloHello.pjs" /u:Hello_VBScript /rt:Hello_VB /run /exit /c IF ERRORLEVEL 3 GOTO CannotRun IF ERRORLEVEL 2 GOTO Errors IF ERRORLEVEL 1 GOTO Warnings IF ERRORLEVEL 0 GOTO Success:CannotRun ECHO The script cannot be run GOTO End :Errors ECHO There are errors GOTO End :Warnings ECHO There are warnings GOTO End :Success ECHO No errors GOTO End :End The batch file calls the TestComplete application and passes parameters including the test suite. Then based on the results (ERROLEVEL), it echoes a status back to the command line. This works well except for the exit code. When the script agent runs this batch file, it will always receive an exit code of 0, unless there was an issue runnig the batch file. A quick and dirty solution is to simply add EXIT # to each ERRORLEVEL section. In this example, "Errors" is mapped to the "Failed" status, "Success" is mapped to the "Passed" status, and "CannotRun" and "Warnings" is mapped to the "Indeterminate" status. This solution does not require changing the source code.
REM Clears the screen CLS @ECHO OFF REM Launches TestComplete, REM executes the specified project REM and closes TestComplete when the run is over "C:Program FilesAutomated QATestComplete 6BinTestComplete.exe" "C:Program FilesAutomated QATestComplete 6SamplesScriptsHelloHello.pjs" /u:Hello_VBScript /rt:Hello_VB /run /exit /c IF ERRORLEVEL 3 GOTO CannotRun IF ERRORLEVEL 2 GOTO Errors IF ERRORLEVEL 1 GOTO Warnings IF ERRORLEVEL 0 GOTO Success:CannotRun ECHO The script cannot be run EXIT 3 GOTO End :Errors ECHO There are errors EXIT 1 GOTO End :Warnings ECHO There are warnings EXIT 2 GOTO End :Success ECHO No errors EXIT 0 GOTO End :End
Suggested enhancements to the Script AgentIf you use TestComplete and feel comfortable working with C++, there are some improvements that would enhance the script agent:
- Have the script agent use the TestComplete status when updating the trjob file (i.e., instead of "Passed", enter "Success"). This would require adding logic to handle more exit codes.*
- Have the script agent simply read the ECHO output and use this to update the status of the trjob file.*
- Figure the reporting in TestComplete and add a reference to the file in the trjob file.