June 2, 2011

Pixar’s Templar Asset Management System

MERGE User Conference

Developers who showed up to the talk by Mark Harrison, technical lead of the data management group at Pixar Animation Studios and the instigator of Templar, Pixar's asset management system, might have been disappointed that 3D glasses weren’t required. They were rewarded nonetheless by a crude, but entertaining, animation of frames going to the render farm at the end of the talk.

Mark Harrison, Pixar
Mark Harrison of Pixar presenting "Don't Let a Bad Trigger Run Your Checkin!

The first customer use case presented at the Perforce 2011 User Conference, Harrison explained his company’s love affair with Perforce and Perforce triggers – and how that love briefly spiraled into an obsession. Starting with one server in 2004, they forecasted needing about 10 depots and servers by 2011. The actual number is 92, he announced to audible gasps from the audience, followed by murmurs of “3D.” Perforce is at the heart of the Templar asset system, which tracks data and metadata for movie, art assets versioned and unversioned file repositories. “We really like triggers -- we think is the second best thing we love about Perforce,” said Harrison. “But everyone’s reaction was, ‘Oh great, can you make us a trigger?’”

The combinatorial explosion of triggers/depots resulted in a giant management headache: trigger breakage and hangs, slow sequential execution of triggers and cross-depot trigger management, among other things.

The solution was to reduce the number of triggers while increasing the number of isolated work queues. “It took a long time to get just right -- two years, three major revisions – but we solved the problem of having lots of triggers by moving all the complexity out of Perforce,” he said.

The other piece of the puzzle was application polling: Checking to see if the trigger code had completed or failed. “That’s not a bad solution for many cases, but we need minimal latency. We have a render farm  with 5000 cores and the last thing we’d want is 5,000 database connections all querying, is it done yet?” The solution included asynchronous background requests.

The animation Harrison played, albeit not in 3D, did show “Templar broadcasting” of diskfarm events to the server farm. “This turned out to be a really useful internal video. We could show people, ‘This is your movie being rendered. Each frame is one of those balls dropping down.’”