August 20, 2014

Lightning Fast Syncs

Traceability

Recently I was working with a customer who was performing performance tests and they were particularly interested in the performance of sync over a WAN connection.

Their test was pretty simple. They had 60 files that were a mix of text and binary filetypes, where the largest binaries were ~10MB and the smallest text files were ~100Bytes. All together the files were 135MB. They had also introduced some 140ms network latency into the test environment. The “sync” time they were experiencing was about 300 seconds.

For those of you that like to skip to the end of the story, we were able to get their sync time down from 300 seconds to 15 seconds. What follows is how we did it:

  1. Configure the operating system’s TCP settings – we used http://fasterdata.es.net/host-tuning/linux/ as a guideline to evaluate the appropriate settings.
  2. We increased the Perforce Server’s send/receive buffer sizes by 8 times. The default is 64K which is really quite low. 
    p4 configure set net.tcpsize=524288
  3. We turned on the compress option in the client/workspace. 
  4. At this point we stopped, took a breath, and ran our tests.
    p4 sync...
    The results of our tests were a good improvement in that our sync time when from 300 seconds to 200 seconds. Interested if we could improve it more, we continued tinkering…
  5. We increased the Perforce client’s send/receive buffer sizes by 8 times.
    p4 -v net.tcpsize=524288 sync ...
    The results of our tests were an incredible improvement from 200 seconds to 44 seconds.
  6. We enabled the Parallel Sync option to our test. 
    p4 configure set net.parallel.max=5
    p4 -v net.tcpsize=524288 sync --parallel "threads=4,min=1,minsize=1" ...
    The results of our tests were another marked improvement from 44 seconds to 15 seconds.

As you can see, there are a number of tunings we did to achieve the spectacular results. Each step by themselves can add big gains. For example, we could have jumped right to the Parallel Sync in step 6, but combining them together produced eye-opening results. Our approach boiled down to a few simple concepts:

  • If you have network latency always check that your OS is configured accordingly.
  • If you have network latency always enable the compress option in your Perforce clients.
  • If you have network latency, increasing the network buffer sizes of both the server and the client.
  • If you’ve got bandwidth available fetching files in parallel outperforms fetching serially.

Some of you may have noticed that we ran all of our tests from the commandline. But these options are available with P4V, too. Using the techniques described at How to Create a Custom Tool in P4V (http://www.perforce.com/blog/140618/how-create-custom-tool-p4v) I created two custom tools that allowed me to achieve the same performance in P4V:

FastSync – increases the client’s receive buffer

FastSync

 

ParallelSync – fetches files in parallel

ParallelSync