June 2, 2011

How NYSE Euronext Scales Worldwide with Perforce

Events
MERGE User Conference

It’s a complex, dynamic production environment that has stringent uptime requirements, is run by a five-person team on 9,000 servers and incorporates software development artifacts and processes from multiple companies. Did we mention it’s highly audited and regulated by the Securities and Exchange Commission?

In the second Perforce customer presentation for the 2011 conference, U.S. head of release management for NYSE Euronext Valerie Hendrickson and global head of release, change, configuration, automation, tools and services Adam Breashears laid out exactly how Perforce makes all this possible.

NYSE Perforce User Conference
Adam Breashears (left) and Valerie Hendrickson (right).

The company fields 198 active software projects and 6,600 production releases per year – and that release count continues to grow. Meanwhile, the business challenges include rapid deployment, six company merges in as many years, increasing code line complexity, pressure to reduce development time to production, a mushrooming server farm due to demand, zero tolerance for system downtime, seamless software rollbacks and traceability of all activity.

Perforce: A Lifecycle Tool?

In short, there’s no room for error. Another aspect that makes NYSE’s massive implementation unusual among Perforce customers is the fact that they use it for production as well as development.

“There’s just one section of the SDLC where we don’t have Perforce: It doesn’t do build for you,” said Breashears. Even there, however, they place every artifact associated with the build in Perforce for traceability.

“We’re always asked, ‘How can you be sure that no one sneaks code into your system? How do you maintain security?’ It’s almost impossible to try to sneak something in, because we check everything every night,” said Breashears. “What’s the algorithm for change comparison?” a curious attendee asked. “It’s Perforce’s algorithm: p4 diff,” he said. When needed, production rollbacks take only seconds across entire machine farms.

NYSE’s philosophy around lifecycle is dual. On the one hand is controlling how its software is used, detecting unauthorized use,  deployment or change. From the more traditional SDLC viewpoint, Breashears talked about how they use Perforce to track creation, QA, release, staging, distribution, utilization and deprecation.

Valerie described in detail the Perforce depot structure. Production deployment requirements include very rapid application deployment, scaling to 9000 servers, the ability to track application files back to source code. “We use distributed global proxies to reduce latency. Our installation time is same whether we are installing one release or many,” said Hendrickson.

Further, the production repository is a meta-map of all production machines, thanks to Perforce’s flexibility around client specs. Spec components pull from different repositories for third-party software packages, release objects, scripts and the like.

In the space of a few years, Perforce has surpassed its initial SCM use case for NYSE, becoming the “root of our software production environment. If you can’t scale the size of your human team, you have to scale the automation they have access to,” said Breashears. “We’re coming from a similar mentality of what you’re hearing today, which is that versioning is a really powerful concept.”