October 22, 2008

Generate Test Runs

Helix ALM
Automation is a key element in ALM. From reducing human error to improving efficiency, a well designed automated plan will improve your current ALM process. This articles contains two versions of a sample SOAP applications that automatically generate test runs in TestTrack TCM. The first version is a stand alone console application. This could be used in an automated build process. The second version is meant to be used with TestTrack triggers. When a defect is fixed, automatically generate test runs for the linked test case. [toc]

Stand Alone Version

Use case

An automated build process is scheduled. If the build is successful, automatically generate test runs for specific test cases. Optionally an automated testing process could also start and update the test runs by the script agent.

The Application

A Java and C# versions are provided. The C# version was created using Visual Studio 2008, so ideally, you should have the .NET 3.5 framework. The Java version was created using NetBeans 6.5.1. The tool takes three parameters:
  • Project Name -The TestTrack Project where the test cases are located.
  • Filter - The filter to use when retrieving list of test cases. Not passing a filter would result in test runs generated for all test cases.
  • Test Run Set - The Test Run Set to use for the generated test runs. If it does not exist, it will be added to the list.
The tool connects to TestTrack and logs in to the project specified in the first parameter. Once logged in, it retrieves the list of test cases that pass the filter provided in the second parameter. For each test case, a set of test runs is generated. All "included" test variants are used, so multiple test runs per test case is possible. The test runs are placed in the Test Run Set passed as the third parameter. If the Test Run Set does not exist, it is added to TestTrack. Errors are logged to an error file.

Configuration

The tool also needs information to connect to TestTrack. This information is read from a configuration file. The configuration file is called CMConfig.xml. The information stored in the configuration file is:
  • TestTrack SOAP CGI URL
  • TestTrack SOAP username
  • TestTrack SOAP password
<TestTrackData>
	<UserName></UserName>
	<Password></Password>
	<TTServerAddress></TTServerAddress>
</TestTrackData>

File Location

In the .NET version, the config file and error log are in the same directory as the executable. In the Java version they have to be in the directory where the Jar file is launched from.

Download

The C# application can be downloaded from here. The bin directory contains debug build of the application and a sample configuration file. The src directory contains the Visual Studio 2008 project. The Java application can be downloaded from here. The bin directory contains a compiled Jar file, all libraries needed and a sample configuration file. The src directory contains the source Java files.

Trigger Version

TestTrack Studio provides tools that allow you to establish such automation processes. From sending e-mail to assigning an item, TestTrack Studio's Automation Rules deliver the tools you need. This article uses a specific aspect of the automation rules: triggers. Triggers can be used to prevent an action, change the value of a field, perform a workflow event or run an executable. This example will launch an executable.

Use case

A test case is defined to test a specific functionality. Test runs are generated and testing begins. A test run fails, and a defect is created. When the defect is saved, the test case and test run are linked with this defect. Once the defect is fixed, test runs are regenerated automatically to test the fix.

The trigger

The trigger is setup to run when a defect enters the "Fixed" state. The action is set to "Run Executable". The executable is actually a script/batch file that calls the script and passes some arguments.

The script

There are two versions of this script. Once is written in C# and the other one written in Java. The C# version was created using Visual Studio 2005, so you will need the .NET 2.0 framework. The java version was created using NetBeans NetBeans IDE 6.1 (Build 200805300101) and you will need Java 1.6.0_07. The java version used the TestTrack SOAP SDK classes available here. Both scripts do exactly the same, in fact, if you put both scripts side to side, you'll see that they are almost mirror images of each other. The script looks at the defect that is fixed, and pulls the links associated with this defect. It looks for any links that are test cases and it generates test runs for each test case. The script currently creates a test run for each test variant combination defined in the test case. It does not look at the failed test run that created the defect to determine the test variants. This could be done, but would make the script a bit more complex. Optionally, the script can also add a parent/child link between the defect and the new test runs. This functionality is not present currently in TestTrack, so it was decided to have it as an option with this trigger.

The Parameters

The script needs the following four parameters:
  • TestTrack SOAP CGI URL
  • TestTrack SOAP username
  • TestTrack SOAP password
  • Whether it should create a link (true creates link)
Each download contains a batch file that shows parameter examples.

Download

C# version

The C# version can be downloaded from here. The ZIP file includes the executable, batch file that launches the executable, and the Visual Studio 2005 project.

Java version

The Java version can be downloaded from here. The ZIP file includes the JAR file, the batch file that launches the jar file, and the NetBeans 6.1 project.