November 9, 2012

Global Development and Git Fusion: Remote Build Farms

Media & Entertainment
Git at Scale

Git Fusion adds another interesting tool to Perforce's arsenal for supporting large scale automated processes, particularly at remote sites. The arsenal already contained proxies and replicas in different configurations, so now you have even more flexibility to keep up to speed with the needs of continuous integration and continuous delivery.

Global Development Scenario

This is an example of how to usefully deploy Git in a global development situation. The concept could easily be extended to use Git for deployment to several application servers.

global development senario

In this scenario, we have a main Perforce repository in California supporting the largest development office. (For the sake of clarity, I'm not showing all the users and build servers that are in this office. Assume I've got a couple of replicas to support them.) We also have two remote sites in Singapore and Hong Kong. Each of these sites uses a very small piece of the main repository.

Singapore and Hong Kong pull from Git Fusion, using views that narrowly define the data they need. They maintain a Git 'site master' repository that feeds their build farms, perhaps through additional Git mirrors.

The Pros

So why not just use a Perforce replica configured in build replica mode? The primary advantage is that Git Fusion allows you to be very selective about how much data you pull from Perforce, whereas a build replica is normally a full copy. If the main repository has terabytes of data, that's an important consideration when you're working on a WAN that spans an ocean. Perforce replicas have some capability for filtering data, but that requires a little more advanced configuration compared to just defining a Git Fusion view.

Also, it's relatively easy to chain several Git mirrors together to service several remote sites. That means that Hong Kong can pull from Singapore, instead of directly from California. That's potentially a big win. Perforce replicas will support chaining in 2013, but again that's an advanced configuration.

The remote sites can still see the big picture, of course: they can pull and share any Perforce data they need. And Git Fusion lets you take advantage of Perforce security and access control.

The Cons

The biggest disadvantage to this approach relates to auditing: If you work in a tightly regulated environment, you may not want to give up the detailed auditing that Perforce provides over exactly which files are synced to each build server.

Also, if your build process writes data back to the main repository, that's trivial with a Perforce forwarding replica, but will require some extra work in this scenario if you use additional Git mirrors in between the build server and Git Fusion.

What Does Your Deployment Look Like?

Are you already trying these types of solutions to support continuous integration and delivery in a global setting? Share your success (or horror) stories! All guest writers on the Perforce blog get some cool Perforce swag. In the meantime, grab Git Fusion and head over to the Git Fusion forum for advice on how to solve your global development challenges.