May 28, 2011

Continuous Builds at Perforce

MERGE User Conference

I'd like to open a portal (my favorite game!) to provide a view into builds at Perforce.  For a while, we had a series of cron driven scripts to do nightly builds for most of our products.  However, with several development groups using Agile and Kanban and the need for testing on a more than once daily but rather cyclical basis, it was time to make the move to automated continuous builds.  We considered our options including building our own framework, but, as we already had our hands full, we went with the Electric Commander tool.  Soon after we attended the Electric Cloud Customer Summit and got some good pointers and tidbits to help us in our approach to automating our builds.

After some dedicated work, we've made the move to automated continuous builds for all of our products, tools and utilities.

Here's how it's set up:

  • Each product has a set of depot directories that is associated with it
  • Periodically (every 5 minutes), a check is done as to if there have been any check ins, that is Perforce changes, made in the associated depot directories since the last successful build
  • If there have been changes, a build is kicked off for all the appropriate platforms (if a build is already running it waits)
  • If there is an error during the build, a build failure email message is sent to the person or people who have checked in changes since the last successful build (as well as to an email list to which other interested parties can subscribe)
  • This loop continues, which means that with new changes, unless a fix has been made, the build may continue to have errors and the build failure messages will continue to be sent out to everyone who has checked in a change since the last successful build

When someone gets a failure message:

  • They will check right away if their change caused the error
  • If their check in caused the error, they will figure out why and check in the fix asap.  They will then monitor that the next build after that was successful.  (We are considering an "all fixed" email that will be sent for successful builds that follow one with an error)
  • And sometimes someone else will help out and fix the issue even if it's not their change that caused the error for various reasons (the owner of the change that caused the error is unavailable, etc.)
  • The build team also monitors the failure messages and will happily help with the fix process (or sometimes it's us, we admit it).

This process works really well because someone who needs a build with a recent change in it such as QA or the developers can get a build fairly soon after checking in their changes.  Also, we find out very quickly if a change has effected the build.

Our next stop -- adding automated testing to the continuous build process.  I'll let you know how it goes.

And in that same vein, I'm looking forward to hearing about some of our customer's build experiences at our User Conference next week.