March 8, 2011

Introducing: Perforce Streams!


If you attended the 2009 Perforce User Conference, you probably saw or heard about the "streams" prototype we were showing around. Since then, the Perforce stream feature has moved out of prototyping and into development. While our development engineers are bringing streams to fruition, I'm taking to the blog pulpit to tell the world about them: What are they? Why are they useful? How will they work? When will they be here? Et cetera. It's going to take several blog posts to tell the whole story, so if you don't find answers to your questions on this page, keep your eye out for the next posts.

Let's start at the beginning...

What are streams?

In the world of version control, a branch is a set of files that is a variant of another set of files. A stream is a "branch with brains."

For example, say my branch is the set of files in //depot/sandbox/laura/ourApp/. You could copy my files into //depot/sandbox/bob/ourApp/, and that would be your branch. Our branches can evolve independently, and we can merge changes between the two.

Perforce has tools to compare and merge our branches, and it's very good at keeping track of what we've merged. But Perforce doesn't actually know what the scope of each branch is. Jane, for example, could branch all of //depot/sandbox/ into //depot/jane/sandbox/, cloning both my branch and your branch into her branch. That's probably not what Jane meant to do. What she wanted was to clone my branch, or yours, but surely not both.

Perforce streams

Perforce StreamsUntil now, however, Perforce hasn't had the metadata to guide Jane toward what she really wanted. It was up to Jane to know in advance what part of the depot held the branch she wanted to clone. And if she got it wrong, she could end up cloning the wrong files. Since Perforce didn't know where our branches were, it couldn't show Jane which files to clone to make her own branch.

Enter streams: A Perforce database object called a "stream" now describes information about a branch -- its location in the depot, its owner, its parent branch, and more (upon which I will elaborate soon enough). Jane can now clone the right set of files simply by telling Perforce which stream to start from. And now Perforce can guide Jane toward integrating changes properly, because it knows how her branch relates to the other branches.

Doesn't Perforce have "branches" already?

I hear you saying to yourself, "Wait a minute -- isn't that what the p4 branches command is for?" True, there is a p4 branches command in Perforce, and yes, you can create a Perforce object called a "branch". But if you've ever tried it, you'll realize that what Perforce is calling a "branch" is actually a mapping between two sets of files.

"Branch" is a tricky term for Perforce. Because we already use it to describe a mapping, it gets confusing when we use it to describe a single, unilateral branch. (The Perforce manuals avoid confusion by using "codeline" to refer to a unilateral branch.) So don't trip on the terminology -- the existing Perforce "branch" object doesn't represent a unilateral branch of files. That's what the new "stream" object will do.

Did Perforce invent streams?

No. Streams have been around for a long time. I first heard the expression from Brad Appleton, in his Streamed Lines paper. I ran across it again in Brian White's ClearCase book. Today, a decade later, a number of version control products feature streams.

What Perforce did do, however, was combine some of its own most distinctive features with the convenience and appeal of streams.

Why Perforce streams?

Many, many shops are already using Perforce for branching strategies that are very similar to what we'll be offering with streams. In fact, the stream model we've designed comes from the ideas our users have shared with us. (I'd name names here, but I'm too afraid I'll leave someone out. All I'm going to say is that many of you -- and you know who you are -- have contributed to the vision.)

So, if you can already do perfectly good branching with Perforce, how do streams add value? In these ways:

  • Streams offer an out-of-the-box branching and merging strategy. Instead of spending your time figuring out your own strategy, you now have the option of using ours. This is great if you're new to Perforce, because you can get started faster. If you're already using Perforce, not to worry -- our stream model is designed to work shoulder-to-shoulder with "classic" Perforce.
  • Streams are effective. The Perforce stream model is based on concepts we've promoted a lot. (See Chapter 7 of my book, for example, or my Google talk.) The feedback we get is that yes, these concepts work. The mainline model, the tofu scale, and the merge-down/copy-up protocol -- all of these make branching and merging less complicated and more agile.
  • Streams give control to project leads. A stream's view determines which depot files are in the stream and how they will appear in users' workspaces. The owner of a stream controls its view. And stream views are inherited, so that when a stream is created or reparented, its view automatically reflects its parent's view.
  • Streams are built on Perforce's most powerful features. Take client view mapping, for example. Files don't actually have to be branched to appear in a stream. Instead, they can be "imported" from the parent stream or from other streams in the system. (I use quotes here, because "import" is just a word we're using to describe the effect. The underlying mechanism is automatic client view mapping.)
  • Streams are easy. With streams you'll never have to edit another client or branch view. For project leads, this means no more coaching individual developers to set up their workspaces -- just set up one stream view and all workspaces for that stream will have the proper view.
  • Streams are pretty. You are going to love our stream visualization tools!

What's under the hood?

The stream feature is essentially a branching and merging application based on the Perforce system. Underneath it all, you're still working with depot files, client workspaces, and all the usual Perforce stuff -- none of that changes with streams.

What's new is that two pillars of Perforce, the Perforce Server and P4V, are doing a lot of housekeeping to make using streams easy. The server, for example, now generates client views and branch views dynamically from stream definitions. It also uses stream lineage to compute sensible defaults for merges and promotions. And P4V is being enhanced to show streams in a slick visual flowchart, and to offer a simple interface for sophisticated stream tasks.

I'll go into detail in upcoming posts. For now, let me just say that this is some pretty potent juju, and I'm looking forward to writing more about it.