Our mission is to achieve complete control of change."
Chief Technology Officer, CCP Games
Halldor Fannar, Chief Technology Officer, CCP
Company: CCP, privately-held
Industry: Independent game studio
Headquarters: Reykjavik, Iceland
Development teams: Atlanta, Reykjavik, Newcastle and Shanghai
Customer since: 2004
Versioning Everything at CCP
Perforce's call to "version everything" is not going unheeded. CCP, makers of the Eve massively multiplayer online game and the forthcoming Dust 514 first-person shooter game for Sony PlayStation 3, has built a new paradigm for large, global game development organization — with Perforce as its hub.
"Our mission is to achieve complete control of change," says Halldor Fannar, Chief Technology Officer for CCP, a privately held firm founded in 1997 with over 500 employees and development teams in Atlanta, Reykjavik, Newcastle, and Shanghai.
Perforce: The Definitive Software Library
What's unique about Eve is that it's all played out in a single universe, with players participating around the world almost 24/7. The company's record thus far is 63,170 simultaneous players out of its global subscriber base of over 400,000.
Though CCP has spread development across the globe, they boast a competitive advantage thanks to the decision to version everything in Perforce; the depot is their Definitive Software Library.
This Definitive Software Library includes:
- Art assets: Audio, textures, geometry, shaders
- Built binaries: Executables, dynamic libs, static data
- External libraries and tools
- Configuration files: Settings, initialization files
- Text: Translations, patch notes
Continuous Deployment in Action
Large expansions to Eve occur at an even clip of every six months, with smaller updates and fixes happening frequently in-between. The game features one of the most advanced character creators in the industry and the players have created over a million characters in the game. Portraits are rendered of these characters for use in the game and on game related message boards. The centralized service that handles the portrait generation is aptly named Paparazzi.
CCP has put continuous deployment in action for this service. Thanks to the tight versioning in Perforce, this important part of the gaming experience requires little manual oversight.
"Before, the Ops team that controlled deployments would always have to remember to deploy the right version of Paparazzi, which was just running on two or three machines in a corner and thus easily forgotten. Now it just happens automagically," says Fannar.
The Paparazzi service wakes up on a regular interval to look for work – that is, to see if any new characters have been created or existing characters changed. However, before doing so the service checks to see if a new version of the character system has been deployed. "Essentially the service is doing a changelist number comparison – it knows the branch to sync from and it can query our database to see at what changelist number our customer-facing character creation system is and compare that to the currently deployed version of Paparazzi. When it detects a mismatch it will do a very efficient Perforce sync to update itself to the right version. Due to how client specs work in Perforce, the server makes sure to only transfer the files that need to be updated.
But there's more, says Fannar: "What's even better is that rolling back is also easier. When you allow your organization to do rapid deployments, allow them to rapidly roll back."
There's a myth that says binaries have no place in source control, Fannar notes: "The usual reasoning is because they take up so much space or that your build process should be perfectly repeatable so you are then storing redundant information."
In fact, executables are just one of several non-source-code assets CCP stores in Perforce:
- Built libraries: "Our software is broken down in DLLs. When you check in source code, build machines start cranking. We store the changelist number and branch, so when you have to go from debugging a DLL in the wild back to source code, it's a breeze."
- External tools and libraries: "We do not require our developers to do any kind of per-project setup. The only way to achieve that is to ship the actual DLLs with the software."
- Static data: An example is celestial data, such as a planet's positioning, radius, and mass, as well as the entire EVE universe.
- Configurations: "This is a big thing with the DevOps movement: you should treat your configurations as code."
The company's efforts to version static data, for example, show how far their commitment to version everything extends.
From 2002 static data was stored, unversioned, in a Microsoft SQL database. As the development activity increased, this became untenable and in 2008 a simple but separate versioning system was introduced into the database. "In 2012, we took a hard line on this and decided we have to move everything to Perforce." In order to make static data mergeable, they used YAML, which stands for Yet Another Markup Language ("It's a real standard, but they didn't really sell it with that name," laughed Fannar.) Additionally, they imposed a number of rules on how the file format is used. For instance, ordered sets is the only list container used – because of its mergeability. With this framework in place they developed a custom merge tool that makes the process even smoother, especially when you integrate it with tools like P4V that allow the user to map a file extension to this custom merge tool.
Now, they use the automated build steps to transform static data in YAML format to an optimal runtime format, which is also versioned in Perforce. The build process also verifies the referential integrity of the data across files and validates that each file is not violating its schema.
But building relational database capabilities into Perforce begs the question: Why reinvent what SQL already does easily? "The reality is, it's more important for us to have the often interdependent code and data versioned and branched in the same system and thus have our developers truly sandboxed, so we've invested in doing this."
Say Goodbye to the 'God Spec'
The company limits branching and uses a mix of replication and proxy brokers to make it all work. As in most game companies, 3D models and textures created by artists in Maya and Photoshop take up large amounts of space, with file sizes averaging 100 MB each. And the artists themselves, typically, think differently about process. "Engineers in general are more inclined to adopt a process, no matter how complex it is, because computer programming teaches you process. Artists and the people that are more on the creative side of development have a different view and in many ways it's a healthy attitude: anything that gets in my way of creation I want to get rid of or intensely ignore," says Fannar.
One of the problems CCP faced when implementing Perforce was defining a versioning process that would work for all developers, including game designers, engineers and artists. "The Perforce visual tool, P4V, is definitely more user-friendly than the command line. But you also have to create a client spec and figure out what folder or branch to sync." At first, Fannar says, CCP didn't explicitly solve the problem. The result was nearly disaster.
"Some people got around the problem by simply creating a 'God spec' that contained the entire depot. Then they would sync the root folder and wait while they're pulling down the entirety of the company," says Fannar. Meanwhile, the Perforce admin was dealing with a crumbling depot and the network admins with a saturated network.
The solution — implemented prior to the introduction of Perforce Streams — was to create automation commands that handle all this management for the user. All the user needs to do to update a branch is to click the sync shortcut for that branch. The process of creating a client spec that fits this particular user is completely automated; artists for instance will require the 'Content' folder and this will be added to their client spec simply based on whether the user is in the 'artists' Perforce user group.
"We started managing all this stuff for our developers. Making changes to the depot structure also becomes easier in this model because we essentially own the specifications and can therefore run update jobs on them rather than having to go on PR campaigns to get people to update their privately owned configurations. Then we looked at the entire promotion model," says Fannar. CCP was able to move from an unstable "branch du jour" model to a streamlined branching model: sandbox > main > staging > release. As a result, released software stability has gone up.
Harvesting Crashes in the Cloud
CCP recently integrated Google Breakpad, an open-source multi-platform crash reporting system, for automated reporting and categorization of crashes in the wild or internally on development machines. The system runs in the Amazon cloud for two reasons: one, because harvesting all game crashes could sometimes occur simultaneously on a massive scale, and two, for reasons of game strategy, Fannar explains.
"EVE players are very cunning. If they find a way to exploit this, they'll do it to gain advantage in the game. If we were harvesting our crash dumps on our internal servers, they would find ways to crash the game at opportune times and slow down the game servers! Yeah, this is how crafty they are."
Now, thanks to the integration with Perforce, a crash report can reproduce the call stack all the way down to the line of code that crashed, with no human intervention, resulting in a vast simplification of what used to entail hours or days of detective work. "Our QA personnel can review this and assign it to the right developer."
A Worthwhile Investment
Thanks to the experience using Perforce as CCP's definitive software library, Fannar advises other users to do the same: "Don't spread your software library across the depot, fileshare folders, FTP sites, and a bunch of DVDs."
A centralized approach, he cautions, requires investment: training, infrastructure, and the dedication to figure out how to transform data and third-party tools so that they can be versioned in Perforce.
Despite all that, "having the definitive software library is a must. As a benefit, you have this rapid workflow for development, debugging and deployment." And, a planet safe for game-play.
All trademarks or registered trademarks are property of their respective owners.