January 10, 2017

DevOps Digest 307: Running Unit Tests with Helix and Jenkins

DevOps

After Issue 306, our basic build scripting tools work. With our NAnt-powered framework in place, let’s extend our continuous delivery pipeline with targets that execute unit tests. It’s nice to have a continuous integration system running a debug build in response to every commit, but all success tells you is that the code compiles. A more interesting measure of quality is whether all the unit tests pass when executed.

Toward that end, we will:

  1. Choose a unit test framework.
  2. Write a simple unit test.
  3. Extend our build scripting with targets to execute unit tests.
  4. Import the test results into Jenkins for reporting and historical trends.

Choosing a Unit Test Framework

Developers have an embarrassment of riches when it comes to unit-test frameworks and tools. Unfortunately, despite its shortcomings, many Microsoft .NET developers never look beyond Visual Studio’s basic command-line utility, MSTest. Unit tests need to run as quickly as possible so that developers execute them frequently. Unfortunately, MSTest runs slower than other tools because it creates folders for every test run.

Developers also need to have a rich assert API to easily trap myriad conditions. This includes a means to check the validity of numbers, enforce type concerns, and so forth. Being able to order tests is a nice, albeit nonessential luxury. Being able to execute in parallel can be a huge win for performance. And simple setup/teardown procedures can be essential for individual and entire groups of tests.

I standardized on the NUnit framework years ago and haven’t regretted the choice. Moreover, NUnit is supported out of the box by JetBrains’ ReSharper product, the only .NET development plugin for Visual Studio I refuse to live without due to its ability to render writing, debugging, and executing unit tests trivially simple. We’ll proceed with NUnit for those reasons.

Writing a Unit Test

To organize the tests, let’s add a couple of new folders to our project’s root. Create a new assembly to simulate a component where code shared by our web application and potentially other assemblies can be developed in a new “Libraries” folder. We’ll put our unit-test assemblies in a new “Tests” folder.

We next create two new class libraries, “ComponentLibrary” and “ComponentLibraryTests”, then manage the NuGet packages for the solution and add the latest version of NUnit to “ComponentLibraryTests”. Note how easy it is to create a simple interface and implementation class that does nothing but return a true value before exercising the implementation with a unit test:

Unit Testing Software Development

Extending the Build Process with Targets

With a working test in place, we’ll extend the build scripting system with new targets to run the test from the command line and generate results in a format Jenkins understands. For this, we’ll add more properties to our build script along with two targets that use them — one to clean test results and the other to run unit tests. Add the following properties to the script:

    <property name="nunit.console" value="C:/Program Files (x86)/NUnit.org/nunit-console\nunit3-console.exe" />

    <property name="test.assemblies.filename" value="DevOpsSampleTestAssemblies.txt" />

    <property name="test.results.path" value="../TestResults" />

The “nunit.console” property specifies where the command-line test runner executable is located. We could have added this to the system path, like we did previously with NAnt, but while lots of projects use NAnt, not every project uses NUnit. Specifying the build scripts is the best way to keep your build environments as simple as possible, making it easy to swap tools.

To make it easy to specify which unit-testing assemblies should be executed, we create a simple text file with one line per assembly. This makes it easy to iterate over each line in the file, as we’ll see later; it also makes it easy to expand the project over time with new unit-test assemblies. In our particular case, the “DevOpsSampleAssemblies.txt” file contains a single line, “..\Tests\ComponentLibraryTests\bin\Debug\ComponentLibraryTests.dll”, which specifies the relative path to our new test assembly.

Finally, the “test.results.path” property specifies where to put the test result files. Putting all the test results in a single folder makes it easier to clean and import them into Jenkins. Toward that end, the first of our new targets cleans that test results folder:

    <target name="CleanUnitTestResults" description="Cleans the unit test results">

        <delete dir="${test.results.path}" />

    </target>

The target deletes the entire test results folder. Executing the unit tests is more complicated. Let’s start with the script:

    <target name="RunUnitTests" description="Executes the NUnit console for a list of test assemblies">

        <mkdir dir="${test.results.path}" />

        <foreach item="Line" in="${test.assemblies.filename}" property="assemblytext">

            <exec program="${nunit.console}">

                <arg value='${assemblytext}' />

                <arg value='--result:${test.results.path}/${path::get-file-name(assemblytext)}.xml;format=nunit2' />

            </exec>

        </foreach>

    </target>

Note first that the “RunUnitTests” target doesn’t depend on the new cleaning target, although this does improve efficiency when it comes time to extending unit tests with code coverage analysis. For now, add the new “CleanUnitTestResults” target as a dependency to the overall “Clean” target.

Second, this bit of script illustrates how to iterate over our simple text file, executing the NUnit console for each line. The “foreach” element offers a number of options, so the “item” property specifies that we want to handle each line independently.

Third, the “result” argument specification demonstrates one of the reasons to prefer NAnt: it brings a lot of easily accessible, built-in functionality to the table. In this case, we’re stripping the file name with a path-related function to specify the name of the test output. Because we named our new unit-test assembly ““ComponentLibraryTests”, this bit of script will cause the NUnit console to dump its output to “ComponentLibraryTests.xml”. In the end, we’ll have one XML output file for each unit-test assembly.

Fourth, adding “format=nunit2” the “result” argument makes it easier to consume the unit-test output in Jenkins. By default, the newest version of the NUnit console produces output in NUnit v3.x, but few Jenkins plugins support that format. We could apply an XSLT transform and import as JUnit, but it’s simpler to output in a more ubiquitous format from the outset.

With these new targets in place, exercise your unit test functionality via the command line:

nant -buildfile:Build\DevOpsSample.build Clean Build RunUnitTests​

NUnit displays the test parameters along with the details of every assembly processed from the output. This information isn’t of any particular use for our automation, so we have some work to do.

Importing Test Results into Jenkins

More specifically, we need to update Jenkins with a new stage in our build pipeline script. The new stage needs to execute the unit tests, using the functionality we just added to the build script, and then import the test results. The following screenshot shows the new pipeline script, which I’ve cleaned up a bit since our last article:

Executing unit tests in Jenkins_pipeline script

The final line of pipeline script invokes the NUnit plugin for Jenkins:

step([$class: 'NUnitPublisher', testResultsPattern: 'TestResults\\*.xml', debug: true, keepJUnitReports: true, skipJUnitArchiver:false, failIfNoResults: true])

The file specification, “TestResults\\*.xml”, tells the plugin to import every XML file it finds in our test results folder. Jenkins is “smart” enough to include a “Test Result” link on its page for any build that includes test results. So all we need to do now is kick off a manual build and click the link.

Unit Test Results

Voila! Our test result for the new build shows a single package of tests: one success and no failures. This data will be maintained as history for our Jenkins project to track the ever-increasing unit test count with each build.

You Ask, We Answer

As previously mentioned, this is your roadmap to creating a successful DevOps pipeline. Don’t understand something? Just ask. Need to dive a little deeper? Send an email to info@perforce.com with your questions. Then, stay tuned for a live Q&A webinar at the end of this series.

Get DevOps Digest Sent to Your Inbox

You don’t need to remember to check back with us each week. Instead, get the digest delivered directly to your inbox. Subscribe to our 25-week DevOps Digest and we’ll get you where you need to go, one email at a time.

See Perforce Helix in Action!

Join us for a live demo every other Tuesday and see the best of Perforce Helix in 20 minutes. Save your spot!