August 29, 2014

For DevOps, Version Control Is Even More Important for Ops than for Dev!

DevOps

Image: slowgogostock via Flickr

I’ve been researching high-performing technology organizations since 1999. For the last two years, I’ve had the privilege of helping design the State of DevOps research study, along with Jez Humble (co-author of the “Continuous Delivery” book) and Dr. Nicole Forsgren Velasquez, in collaboration with Puppet Labs.

One of the most startling findings this year studying over 9,200 survey responses was that version control was consistently one of the highest predictors of performance. And furthermore, that version controlling the production environment was an even higher predictor of performance than version controlling code!

Let’s first talk about performance. We found that high performers were massively outperforming their peers, sustaining:

  • Higher throughput
    • 8x more frequent production deployments
    • 8000x faster deployment lead times (i.e., time required from “code committed” to “successfully running in production”)
  • Higher reliability
    • 50% lower change failure rates
    • 12x faster service restoration times when something went wrong (i.e., MTTR)

Our research also found that those high performers also had organizational performance, being 2x more likely to exceed profitability, market share and productivity goals.

We identified five cultural and technical practices that were the top predictors of performance, including use of peer-review to scrutinize changes (instead of external approvals), version control of all production artifacts, proactive monitoring, high trust organizational culture, and a win-win relationship between Dev and Ops. (The presentation we did at the 2014 Velocity Conference can be found here.)

For me, the biggest surprise was the prominence of version control as one of the top technical predictors of performance. Version control was consistently one of the top two predictors of deployment lead time, deployment frequency and MTTR. It was also a predictor of high job satisfaction.

Top Technical Predictors Of Performance (2014 State Of DevOps Report)

Furthermore, we found that version controlling production artifacts was a high predictor than version controlling code. This is one of those delightfully startling findings that seem initially absurd, and reveals profound insights.

Why would version controlling production artifacts be a more effective predictor of performance than version controlling code? Here are my favorite hypothesis:

  1. There are more configurable settings in the environment than in the code, any of which when misconfigured could result in the service not functioning correctly. (Anyone who has performed code migrations in ERP systems, such as from SAP or Oracle, will recognize that when migrations fail, it’s most often due to misconfigurations in the environment, not problems in the code).
    The countermeasure is that all production configuration must be checked into version control, serving as the single source of truth that everyone can depend upon for all environments (e.g., Dev, Test, Prod, etc.).
  2. Continuous delivery requires that developers be able to create production-like environments on-demand via an automated build process. And of course, to ensure repeatable outcomes, the build mechanism should be checked into version control, as well as all its dependencies. This ensure that anyone can re-create environments based only on what’s in version control, as opposed to individual or tribal knowledge, or worse, in a build document or Wiki page.

(Interestingly, these two behaviors are both listed in the seminal presentation “10+ Deploys Per Day: Dev and Ops Cooperation at Flickr” by John Allspaw and Paul Hammond.)

When Ops use the same version control system as Dev, anybody can reproduce the production environment based only on what’s in version control, and Dev and Ops can depend on version control ensure that changes to the code and environment are continuously being integrated and deployed into the production environment, resulting in high IT performance and organizational performance.