Hidden Path Entertainment
Michael Austin is CTO and co-founder of Hidden Path Entertainment an independent video game company.
Hidden Path develops interactive entertainment software for current and future generations of video game consoles and other computing platforms. Currently, Hidden Path is working on both independently-funded and publisher-funded projects, developing fun and innovative titles for some of the most popular companies in gaming.
Hidden Path Gets to the Next Level with Perforce
At the office-in-the-garage stage, many startups build on a foundation of open source tools. Such was the case for Hidden Path Entertainment. When the company launched in 2006 from the Bellevue, Washington, home of CTO and co-founder Michael Austin, they started by using Subversion for version control. Two years later, the company released the Defense Grid: The Awakening, a tower defense video game that has sold over half a million units to date. Success breeds success, and Hidden Path soon found itself collaborating with other game publishers, including Valve Software.
The Free Lunch is Slow
"We went with SVN because it was free. When you're small, cost is very important. We were on Subversion for about four years. Collaboration worked well, but it just didn't scale. We were using Tortoise SVN for Windows Explorer, but once we were dealing with millions of files, tree scanning could lock people's machines up for minutes at a time," says Austin.
It's not uncommon for even a small game development company to have millions of files in its source control depot. Austin began searching for alternatives. "We were looking at what people were saying about cases where SVN would have problems, and we were one of them: millions of tiny files checked out by lots of people." With the system already slow, not showing signs of scaling well and unstable for backup, Hidden Path "needed to find something better." In 2009, the decision was made easy. "When we got the contract with Valve Software, they recommended Perforce," says Austin, who also wears the hat of system administrator.
Speed? I Am Speed
The company made the switch to Perforce—and not a moment too soon, as it also gained 20 million more files when it began working with Valve. The performance improvement was palpable. According to Austin, "when we had a big project, for somebody to do a get of about 120 GB and 5-6 million files, it would take about two hours. When we switched to Perforce, that get took less than 5 minutes."
One reason for the speed has to do with SVN architecture. While Perforce software version management uses the file system to store files and revisions alongside a database of metadata related to changes, SVN uses the file system itself as the database. "When file metadata is in a parallel file stored in the file system, that adds a lot of overhead to the host. I like that Perforce keeps metadata in a database rather than a million small files. It's a good model."
As a company of 37 developers, Hidden Path outsources a number of functions, including IT. And a recent experience hammered home the value of Perforce when an IT contractor deleted key files.
"A year ago, I got a call saying ‘Perforce is down.' Usually that's because something's wrong with the hardware so I go on the machine and I couldn't see any executables," recalls Austin, who quickly went into panic mode. "It turned out our IT contractor had uninstalled our source control system, in the middle of the day, while people were using it. I was prepared for the worst, but decided I would try just reinstalling Perforce. It picked up the databases and it just worked. That never happens in IT. Credit that to elegant design," Austin enthuses.
"The decision to avoid the registry was brilliant. Perforce was written to be as lightweight as possible, so it puts everything it needs in one directory. There are no hidden files scattered around my computer. When you rerun it, it tries to be smart and looks to see if there's an existing Perforce installation. That's the rare part for me, to have software that works that well."
Thanks for the Support
Today, Hidden Path's entire Perforce database is roughly 600 GB, with 70 people accessing the depot from across many time zones—and no performance issues. "I've seldom seen something scale so well."
Hidden Path has continued to evolve their Perforce usage, emphasizing a need for stable builds, better branch design, proxy servers and shelving, for example. That last item, in fact, dovetails with Austin's praise for Perforce support and product updates. "We expressed a need for shelving. They said they had been working on it and were glad to get feedback. A few months later it was in an update and we were using it."
Austin admires the speed and quality of Perforce's technical support—yet another way the product makes his job easier. "You're talking to the part-time system administrator," admits the CTO. With Perforce, "it's very part-time." Contrasting Perforce to his previous software version management experience, Austin explains, "Subversion support was just Google and tears. Perforce support is awesome. I have only had good experiences with them."