June 6, 2011

Why RIM Gives Perforce Extra Credit for Scalability

MERGE User Conference

Nothing cements customer loyalty like coming through for them in a pinch. That’s what happened when Perforce support and development was able to implement a reference solution for Research in Motion (RIM) in just two weeks.

At the 2011 User Conference, Shannon Mann described his five-year odyssey 

of improving performance after his company’s use of Perforce grew to embrace automated parallel builds and dramatic increases in continuous integration. By 2006, facing severe performance problems and having exhausted all hardware options, Mann’s team made the decision to split their depots.

Multiple challenges loomed, but one was a showstopper: Perforce integration with the bug-tracking software was simply not scalable. “On the Perforce side, it can do 100 jobs per minute. On the bug-tracker side, it’s 12 per minute,” said Mann. With 9 million commands per day across multiple Perforce depots, delays of several seconds per command quickly add up, and the inability to speed up bug tracking looked to derail the project.

“The solution was something we called job content synchronization, or JCS. We would abstract out the process with some type of middleman depot that would leave the existing bug-tracking integration unchanged.” He called Perforce support for assistance, and then began the nail-biting process of wondering if they’d come through for him. After two weeks, he met with Perforce. “They said, ‘We loved your idea so much, here’s a solution.’”

As before, the stand-alone depot has P4DTG integration with the bug tracker. It uses P4Broker to redirect p4 jobs, P4Auth to make the redirect transparent, P4Change to make connections unique, and triggers to request initial job data. Though the complexity of the solution did increase and he acknowledges they still need to develop a disaster recovery plan (which may be ameliorated by 2010.2’s ability to do file replication), JCS allows almost linear scaling of bug-tracking interconnects to multiple depots.

“Now the bug tracker is entirely unaware and unaffected by depots on the other side. The whole thing has been a complete and massive success,” Mann enthused. “Perforce support are your friends; seek their advice. But more than that, this story is actually about how thoroughly they support their own product. They added key details to the JCS solution and they responded extremely well to emergency needs.”