May 23, 2018

When Fast Isn't Fast Enough

Performance
Version Control

Speed and performance are crucial in software development. Reliable speed and efficiency enable global development teams to maintain aggressive schedules and improve productivity. Ultimately, high-performing systems help development teams achieve their end goal: shipping quality products faster. This is why our engineers have invested tremendous time and energy over the last several years to make Helix Core (P4D) even faster and more scalable. Let’s examine some of the performance improvements we’ve delivered over the last few years:
 

Helix Core Performance Improvements Since 2014

Server Improvements

In 2014, we introduced lockless reads, a major improvement to the database architecture of the Helix Core server. This enhancement improved scalability by using resources (such as CPU cores) more efficiently. Lockless reads also helped reduce the likelihood that server requests would be blocked (resulting in waits) on metadata locks.

Dramatic Throughput Increase

At the 2014 Perforce Merge user conference, engineer Michael Shields presented the results of a simulation conducted in our lab that compared performance of the pre-release server with the new updates in the server. The simulation tested the following common scenarios:

The results were dramatic. With the new updates, the percentage of blocked commands decreased by about 92 percent. This means that among the 256 developers who were submitting commands simultaneously, only five of them were waiting, rather than 64. That's a 12 times increase in throughput. 

It is difficult to benchmark the exact performance gains a particular installation of the server might achieve because of all the variables involved. But customers have universally reported better performance after implementing the Helix Core updates that incorporated this innovation.

Automating Builds

In 2015, Helix Core users were increasingly adopting Agile and Continuous Integration (CI). Adopting CI increases the volume of server requests and the amount of data (files) being moved, which puts increased load on your version control tool.

To solve this issue, Perforce released Helix Core 2015.2 with the ability to configure read-only client types. A read-only client reduces the load on the server because the code change is stored in the read-only client for the build and then is automatically deleted.

This feature makes CI faster because the read-only client has its own db.have table. This means all files used for the CI build can be stored in the read-only client’s db.have table and not in the server’s. In addition to decluttering the server’s db.have table, read-only clients improve build performance by automating and isolating build requests.

Check out this Knowledgebase article to learn how db.have improves performance.

Federated Architecture

Helix Core federated architecture was introduced in 2013. Since then, it’s been enhanced on a regular cadence. Federated architecture, which is also often referred to as commit/edge, offers numerous performance benefits.

The biggest benefit of federated architecture is that it allows global teams to practice CI and collaborate as though everyone is in the same room.

With the 2016.1 release, the performance for parallel 'p4 submit' through an edge server improved by batching file transfers. A customer reported that it previously took them six hours to clone a repo to a workstation in their India development center. With the performance improvements in 2016.1, it takes minutes. Developers can integrate their code changes faster, increase the frequency of their commits, and as a result, launch quality products faster.

In the Helix Core 2017.1 release, enhancements were made to improve the transfer of files over high-latency networks. As a result, processing speeds are 16 times faster than in 2016.2, including with federated architecture deployments.

Plus, we see similar performance improvements for communication between any server type: replicas, commit/edge servers, and proxies. This involves TCP connection changes to improve performance over long latency connections. These enhancements work on both Linux and Windows. In addition, receive buffer sizes for the client API was increased to 1MB, from 32K, bringing major improvement for several wide area network operations.

Support for WAN Technologies

The 2017.2 release of Helix Core also bolsters performance across remote sites thanks to support for Aspera, a third-party high-speed transfer solution. This integration prioritizes Perforce file transfers and ensures remote sites are in sync with the central servers at all times.

Aristocrat Leisure Limited, a worldwide leader in gaming technology, is leveraging Aspera and Helix Core to maintain its aggressive schedule and improve productivity. Aristocrat, who transfers terabytes of digital assets across multiple locations, has seen 14 times faster performance with large file transfers. Transfers that once took days or were completely untenable using traditional technologies now only take minutes or hours, all with complete, built-in security for data transfers and zero restriction on the file size or the number of concurrent transfers or users.

Faster Code Reviews and Collaboration

Helix Swarm, the code review and collaboration platform that comes with a Helix Core license, received significant performance improvements during the 2017.2 release. The speed at which reviews are delivered to the user’s browser has improved dramatically. Loading 100,000 reviews now takes about one second to display, compared to about 15 seconds before the release. This improvement is extremely meaningful to many of our customers who routinely manage upwards of 185,000 open reviews from a single Swarm instance.

Failover in Disaster Recovery Scenarios

The 2018.1 release of Helix Core provides support for advanced failover and replication strategies. Failover is now significantly faster due to multiple steps being removed from the process via automation and a new command.

The command 'p4 admin end-journal' has been added to allow an admin to instantly and reliably end a journal being replicated to a standby server via 'journalcopy' at the last replicated consistency point. 

Summary

We have been able to accelerate the performance of Helix Core by listening to our customers, looking at data, building, and testing.  Then we iterate — taking what we learn and applying that to invent new ways to achieve ever-greater levels of performance.

Beyond the product enhancements we talked about here, there are many exciting performance-enhancing topics to review in the “Tuning for Performance” section of the Helix Versioning Engine Administrator Guide: Fundamentals (2018.1).  

Get Latest Version of Helix Core

On an older version of Helix Core? Leverage the latest performance enhancements when you download the latest version.

Not Using Helix Core?

You can reap the speed and scalability features of Helix Core for free! Helix Core is free for up to 5 users and 20 workspaces. Download now.