September 24, 2014

Pixar’s Templar: Big Data Asset Management Built for Scale

MERGE User Conference
toy story
Image: via Pixar

Pixar explained at MERGE 2014 how they manage large assets using Templar, a proprietary asset management system built to handle all their studio data and metadata. It’s used for feature films, shorts, and all other projects because it manages artwork, scripts, movie frames, simulation details, project management data—everything Pixar needs.

What’s more, Templar was planned for a fifty-year timeframe, aiming to guarantee that all their assets will be accessible until 2054. So how do they achieve this, given the incredible scale at which they work? The key to Templar is its federated architecture, which is loosely coupled with software hooks into their workflow pipeline. This allows pieces of the system to be upgraded incrementally for different software and file formats over time.

Unsurprisingly, Templar uses multiple storage mechanisms to meet different goals. Temporary caching is done using large, fast storage on the NFS file system, while scalable storage needs break down into two categories. A proprietary system provides storage for non-revisioned files, but anything that has to be versioned (and that’s a lot) is stored in the Perforce platform.

Pixar chose the Perforce platform because of its ability to scale to any scenario along with its triggers and integration hooks. Pixar was able to build their rendering farm and other tools around the Perforce platform to satisfy their considerable needs for art, technology, business, tools, and everything else. Let’s look at some crazy numbers:

  • Largest Single File: 900 GB
  • Largest Single Checkin: 6.5 TB
  • Largest Single Depot: 47 TB
  • Total Perforce Storage: 160 TB

It’s always gratifying to hear customer success stories, but it’s downright amazing to hear how Pixar has leveraged the Perforce platform at such an enormous scale.