April 22, 2013

Pixar at Merge 2013: Big Data at the Movies

MERGE User Conference

toy storyPixar is living the “version everything” dream — in technicolor. Via Perforce, Pixar not only tracks movie data and metadata, art and other assets, it triggers rendering of each animation frame on a massive server farm with thousands of cores. Now nearing a decade of passionate Perforce usage, the company is exploring how to manage and mine its big data using enterprise versioning services.

The Emeryville, CA-based animation studio was put on the map with Toy Story in 1995. By 2004, it had implemented its first Perforce server and forecasted that seven years later they’d need 10 depots. At the 2011 Perforce User Conference, however, Mark Harrison, technical lead for Pixar’s data management group revealed, to audience gasps, that the company had 92 Perforce servers. Two years later, the size and number of depots have continued to grow, but the big reveal — by how much — will come at the Merge 2013, The Perforce Conference, when Harrison and fellow presenters Mike Sundy, digital asset system administrator, and David Baraff, senior animation scientist will provide an entertaining look at how Pixar has coped with this explosion.

"Perforce is great for big data — even for data that won't be revisioned, branched, etc. Building a system on top of Perforce has been a good experience,” says Harrison. "Whoever at Perforce thought of ,d storage is a genius beyond imagination."

In addition to encouraging others to use Perforce to version big data, the trio will unveil their custom archiving and deduping solutions, aptly named the Underminer and Shrink Ray. “We feel we are getting good results in some new areas such as big data and hope other organizations can benefit from our experience.”