Memory Leak Testing in Continuous Integration Systems

Continuous Integration (CI) is the process of frequently integrating changes from development into the existing code repository, which automatically triggers a build and test of the changes.

In CI systems, the code testing process is automated. It typically includes unit tests, functional tests, and, sometimes, performance tests. However, none of these tests analyze the impact that new snippet of code will have on the application’s memory usage, i.e. memory leak testing. Indeed, modern quality assurance testing of software most often does not include testing for problems with the utilization of memory; the reason for this is in the lack of testing solutions which include a view of memory.

Memory testing involves validating a C or C++ application's use of memory and looking for memory leaks or illegal management of memory, such as buffer overwrites. Memory leaks can be catastrophic to an application, resulting in hangs, buffering, or crashes. In the worst-case scenario, they are reported by a customer.

While developers may routinely run their own memory tests on the code they are actively developing, memory testing later in the development cycle is often reduced, or overlooked entirely in the rush to deliver a new release.

Let’s take a closer look at the benefits of adding memory leak testing and memory usage analysis to the CI process.

Advantages of Integrating Memory Leak Testing

Continuous Integration systems enable constant feedback of the impact of changes on a codebase. Using the automated capabilities of a CI system to conduct routine memory-use analysis delivers the same advantages as it does for regular functional testing:

• New code changes are immediately tested, making them easier to fix.
• Any new illegal uses of memory and memory leaks are detected and reported through the CI system.
• The CI system’s test reports provide transparency to a problem that is often undetected until it is too late.
• Generation of memory result files allows for follow-on detailed analysis of the application's use of memory.

4 Steps for Testing Memory Leaks

From a high level, any changes to a codebase monitored by a CI system, such as Jenkins, TeamCity, or GitLab CI, results in the execution of the following common steps:

1. Build the Application

Changes to the codebase are gathered and the CI system kicks off a job to build the application. If the application is built successfully, another job is triggered to regression test the application. If a build failure occurs, the CI system records information about the failure in the job build report and notifies the team. Testing of the application will not be done if a build failure occurs.

2. Regression Test the Application

With a successful build of the application, the CI system starts a job to test the functionality by running unit, integration, and system-level tests. Again, test results are gathered by the CI system and presented as part of the results of the test job and reported to the team.

3. Memory Test the Application

Using a tool, like TotalView, to test the memory usage of an application under the CI system is not much different than regression testing the application. It usually involves running the test suite again but, this time, running each application test under the control of the memory debugging technology. Depending on the test suite, this might be as simple as adding an argument to the scripted call.

4. Generate Leak Detection Report

The CI system gathers the test results of each job run and presents them as part of a report for the team. When TotalView memory tests are run, TotalView generates a test report log and, if instructed, memory result files. The CI system can be configured to pick up the log file as part of a post-build action and sent to team members. Application run logs also provide details on the location of the log files and memory result files so that they can be examined.

Read a more detailed example of How to Test for Memory Leaks in Jenkins. >>

Tool for Testing Memory Leaks

CI systems provide many benefits for simplifying the build and test process, shortening development cycles, and improving code quality. Integrating TotalView leak detection and memory usage capabilities into the CI process helps teams find and eliminate memory problems as a routine part of the CI process – and before they get noticed by your customers.

Try TotalView Free

Further reading

• Learn about using tools to secure and optimize processes at scale with Perforce's enterprise automation 101
• What Is Continuous Integration?
• Memory Testing in Software Development
• How to Test for Memory Leaks in Jenkins
• How Jenkins CI Parses and Displays JUnit Output
• JUnit Plugin
• xUnit Plugin
• What Is Regression Testing?

Note: This post was originally published on August, 17, 2017 and has been updated for accuracy and comprehensiveness.