December 7, 2016

Perforce Helix for Game Developers – A Distributed Single Source of Truth


I recently read a blog post from Atlassian on why you should switch to Git (and specifically BitBucket of course) for game development. Since the post mentions Perforce explicitly, as the incumbent industry standard for game development, I would like to respond in kind and clarify a few points. A first question is: why do the leading game development studios in the world use Perforce Helix for their version management?

Project Sizes

Video game development is a funny business. Writing software usually involves potentially large numbers of what are essentially text files of a reasonable size. But video game development adds to that a great deal of binary assets: raw artwork, “baked” textures, models, animation, sound, pre-rendered cinematics, and so forth.

Versioning these files is a serious challenge, and not just because they’re binary and often impossible to merge. The files are often very large as well. Many modern video games can include north of 50 GB of such final data. In fact, these binary files comprise the majority of game content, often 90 - 95% of the total number of files.
The total assets for a typical AAA game these days can exceed a terabyte. That is your head revision and these numbers are true for a standard HD (1K) game. And of course, this is old tech: many studios I have spoken to are now working on 4K games (see the recent availability of the PS 4 Pro), which can significantly grow such assets.
You might argue that the majority of games these days are written for mobile phones or browsers, but even here the screen resolution is going up and—ignoring retro games for a moment—customers demand games that can make the most of their shiny new screens, thereby ramping up those asset sizes as well.
It’s not the terabytes of disk space for the head revision that frightens me when I have to spec the hardware to store all this stuff, it’s the overall size of history (100s of TBs) and the rate of change, often exceeding 50 GB a day and sometimes well over 500 GB a day. That’s a lot of data that has to be synced for every single contributor when using a tool like Git, which doesn’t offer narrow cloning or other such features.
Which brings me to team sizes. Even today you can still create fantastic video games with less than a handful of people, but triple-A games require teams of hundreds of artists, developers, and other contributors. Much of the time it’s impossible to bring that much talent together in the same location, so geographically distributed development is increasingly the norm.
Now imagine a single Git repository using Git-LFS to store all the large binary assets. You can avoid much of the pain of Git storing all the history of these binary files using Git-LFS, but what you cannot avoid is that every time you refresh you will get hundreds of GB of asset updates, and you will very quickly saturate your network bandwidth (and the patience of your developers) this way.


Artists and Git

There is an additional issue with Git-LFS: most artists I have spoken to in the industry do not like it. Artists can often be a bit of a paradox: immensely creative and meticulous in their attention to artistic detail, but often not very interested in the technical details of version management.

Git is a great tool for developers, and I use it myself almost every day. I understand and appreciate the underlying data structure, and I can handle many of the arcane commands it comes with (marveling at the ingenuity of coming up with a different option for every command for the same logical operation, as it were).

Artists, on the other hand, don’t want any of that. In fact, some of them are reluctant even to use version management at all, preferring file name suffixes like “v1”, “v2”—begrudgingly committing final work into version control  if at all. And when they do use version control, artists prefer a graphical interface to the command line, preferably one with the ability to show thumbnails and differences between versions of images.

If Git is the only option, many smaller studios choose to use a separate tool to version their assets, typically Subversion or even Dropbox. The result is that contributors become separated into silos and that is bad news.


The Need for a Single Source of Truth

One would think that assets, game code and game engine have nothing to do with each other and can happily live their independent lives. In reality, tracing a bug down across several unconnected repositories is a problem. If that bug broke the build or (even worse) made it out into the wild, a lack of cross-repo traceability is a pain that game designers, QA and build managers can do without.

We have many testimonials on our website from game development studios who have gone through the same process of bringing all files of a game into the same repository, a “single source of truth”. The advantages are obvious, as the Atlassian post recognizes as well.

What the Atlassian post did not mention is that a single Git repository rapidly grows far too unwieldy, which inevitably leads to Git sprawl: breaking that repository into smaller chunks. Which of course leads right back to separate silos, the stitching together of which is a task not easily solved with Git submodules.

Now let’s compare this with Perforce Helix. I can have a single codeline that holds all game assets and code together, yet still break out individual chunks into my local workspace exactly the way I want to work. Programmers typically don’t need the original raw art, audio, or other assets, only the “baked” versions from the build pipeline for testing. Similarly, artists and other non-programmers can focus only on the small subset files they actually need without cluttering up their machine, and without being held hostage while their machine syncs large content they don’t need.

Artists also enjoy the benefits of a central server, such as file locking to avoid overwriting others’ work. I also find it useful to see if someone is working on the same file as me, even if the file can be merged later, as this avoids nasty surprises when it comes time to submit my work.

Helix also supports DVCS workflow and even native Git. Developers can create Git repositories from the centralized codebase using Git Fusion if they prefer, or use the Helix DVCS features. Users can also manage their large repos better with narrow cloning, breaking sprawling source code in to smaller chunks. Helix even supports shallow cloning as part of the native toolset. The changes they push are submitted into the same “single source of truth”. This is even true if you are an artist as Helix DVCS allows you to lock a file in the central server (as long as you can reach the network) even if you’re working with a local repository.

Many game development studios have more than one office, and some are spread around the world. As industry titan Electronic Arts (a long-standing Perforce user) put it: there is simply not enough talent available anymore in one single place to create a triple-A game. All the different offices must contribute to and work with the same codebase, a geographically distributed “single source of truth”.

This is exactly what Perforce Helix provides. Thanks to Helix replication technology, users can work against a local replica at LAN speed while their changes are synchronized around the world over WAN connections. This enables any number of geographically disparate locations to work together as a single worldwide team.

But don’t take it from me. I recently had a series of interviews with different game studios who standardized on Perforce.


Why Should I Get Started with Perforce Helix Right Away?

From my statements above you might conclude that Perforce Helix is useful only to large, AAA-game development companies. But in fact, we regularly get requests from start-ups and small (e.g., 5 - 20 users) companies all the time. There are several reasons for that:

Familiarity. Just as Git is a frequently used tool in software development generally, Perforce Helix is the standard for video game development. This means if you are hiring an experienced games developer, chances are that person already knows and loves Perforce.

Integrations. All major game engines like Unity, Unreal, CryEngine, etc. have integrations with Perforce Helix (and many were built using Perforce as well). This is equally true of graphical tools like Maya or Photoshop. Beyond its broad suite of integrations, there are also a whole range of APIs should you need to go beyond existing integrations.

Speed and scalability Perforce Helix is known as the fast version management system. You will occasionally hear from video game developers that Perforce is slow, but what they often forget to mention is the magnitude of data (e.g., downloading terabytes of data). Performance is key for games development as is scalability: your assets will inevitably grow is size as your game develops, and you do not want to find out that your tools cannot cope with the increased size while fighting to deliver a milestone or ship your game.

Last but not least, most small studios dream of becoming big players in the industry. Isn’t it reassuring to know that you invested into a tool that can grow with you all the way to the top? Or would you prefer reworking your entire tool chain and practices with each new success? We have a wealth of customers who have made this journey, as you can see from the testimonials on our website.

Perforce Helix remains free for small teams. Download the product and speak to us if you want to learn more about how games studios are successful today with Perforce Helix.

Happy hacking!