December 5, 2016

DevOps Digest 304: Creating a Development Stream


Having chosen a high-level branching strategy, and made the choice to use Helix streams to help implement it in DevOps Digest 302 and 303, let us return to our previous question: what do we want to build automatically?

It would be easy enough to automate building the project that already works. And for small projects, letting everyone work in the trunk and building directly from it with every commit is fine.

But this isn’t best practice for non-trivial projects. If everyone works in the trunk, then every bad commit likely inconveniences everyone on the team and breaks the build. Beyond the obvious inefficiency, it also foments distrust and negativity among team members, particularly when some individuals are sloppier in their testing than others. A much healthier practice is to divide up the work using child streams, delivering units of work to parent streams only when the work is ready.

This has ramifications on what we build in Jenkins, how many projects we create in Jenkins, and how we organize them. There are a few different best practices we can recommend, using them to guide our own work.

First, most organizations want to separate development builds from release candidates (or, QA builds) from releases. Our streams, and the Jenkins projects we use to build them, need to fit this pattern to illustrate the point.

Speaking generally, you want to do development builds on every commit to catch errors quickly. As such, an incremental build strategy, effectively what we’ve already created in Jenkins, is the right approach for sake of speed. That way, we can speedily sync, build, and run basic tests on our project. Unfortunately, errors can go uncaught if we aren’t careful about cleaning up our build environment before each run. Ideally, keep build artifacts completely separate to make avoiding these errors easier, but that’s not always simple.

In contrast, you typically want to build release candidates only as often as required to validate that the current product passes more exhaustive automated testing for a full-up release. This is the difference between simple unit tests and more thorough integration tests. Release candidates effectively become releases once they pass the required testing and other organizational gate checks, so this is the place where quality assurance spends the bulk of its effort. For these builds we’ll need something different than what we’ve done. We’ll want to get a fresh copy of all the source to catch any errors that can otherwise slip through the cracks.

We clearly have some work to do over the next chapters including:

  • Changing how we develop with our sample project
  • Changing how we do things in Jenkins accordingly
  • Implementing unit tests, code coverage analysis, and other metrics to optimize our pipeline

So let’s start by creating a new development stream. This will provide developers a single stream for delivering their work, distinct from where QA and other contributors work. With that new stream, we can update our existing Helix workspace to use it and rework our Jenkins project to build automatically in response to every commit for a wonderfully fast turnaround.

To create a new development stream, switch to the Helix Visual Client and check that the workspace we created is selected in the workspace tab.  Then switch to the streams tab, right click the existing mainline stream, and choose the option to “Create New Stream from ‘Main’….” on the resulting context menu. I’ll name this new stream “Development” for sake of illustration, but use this level of streams to separate work however makes sense.

In particular, note that I’ve checked the box labeled “Branch files from parent on stream creation” at the bottom of the screenshot below. It’s not a critical necessity, but it’s a nice time-saver that I’ll leverage for this process. Otherwise, populating the stream takes a second step, which isn’t useful in this particular case.

Creating a new development stream in Perforce Helix

Now that we’ve created the new development stream, the Helix Visual Client updates its graphic representation in the streams pane. The two arrows between “Main” and “Development” below indicate that changes flow in both directions. This is typical (and the default) for less stable streams delivering their content to more stable streams.

development_branching_streams pane

Both arrows are also the color gray right now. That’s because we checked the box to branch on stream creation and this indicates that there are no changes ready to flow in either direction. Had we not checked that box, the downward arrow would be green, indicating that changes in the “Main” stream were waiting to be delivered to the “Development” stream. We’ll learn more about those arrows later, but suffice it to say for now they make it easy to see where work is waiting to be delivered — a life-saver in complex development scenarios.

To use our new development stream, we need to update the workspace we created earlier. To do this, switch to the workspaces tab, right click the existing workspace, “DevOpsSample-Main”, and choose the option to edit. Click the “Browse…” button near the stream selection (which says “Main (//DevOps/Main)”) and choose the new Development stream in the resulting dialog.

Updating our workspace in Helix Versioning Engine

Click the “OK” button on that dialog to choose the new stream, then click the “OK” button on the workspace dialog to save the changes. Dismiss the prompt by clicking the “Update” button, reading the text before you do as it can be a little confusing for new users.

Updating your Helix workspace

In essence, whenever you change the definition for a Helix workspace, it means the content you already have is likely no longer what you want. In our case this isn’t true, as we just created the new child stream, but workspaces are such an incredibly powerful abstraction that this is rarely the case. So the Helix Visual Client asks if you’d like to make the most minimal changes necessary to reflect the new workspace view.

Note, that does not include updating existing content or getting new files; the only changes it’s offering to make right now are (1) removing files no longer in use, and (2) moving files that have been mapped differently. It’s trying to make sure you have the right structure to receive the content the workspace specification now entails and don’t have a bunch of old stuff left lying around.

Once we’ve dismissed that dialog box, choose the “Dashboard” option from the “View” menu, to view the content we’ve yet to receive. Dismiss that dialog box and choose the “Dashboard” option from the “View” menu to see the content we’ve yet to receive. In our case, that’s 80 files. This could be confusing until you realize those are the variants we branched from the Main stream into the Development stream. In Helix, a branch is a variant of a file, which retains a common ancestor and information for merging and other operations. This is a significant distinction with Helix compared to other systems, which is why it’s worth laboring.

Helix_workspace dashboard

The dashboard shows you what needs to be done with the currently selected workspace. In this case, we have a link to “Get latest revisions (80 files)”. When we click it, we’ll get another dialog. This is the standard dialog offered by the Helix Visual Client for sync operations. Click the “Get Revision” button.

Helix Versioning Engine_Syncing Operations with New Development Stream

After the work completes, there are no more updates to receive, and the workspace view in the left pane shows our local copy populated with the files we branched into the Development stream.

Join us next week for DevOps Digest 305: Automating the Debug Build when we make a few changes within Jenkins to account for the work we completed today.

You Ask, We Answer

As previously mentioned, this is your roadmap to creating a successful DevOps pipeline. Don’t understand something? Just ask. Need to dive a little deeper? Send an email to [email protected] with your questions. Then, stay tuned for a live Q&A webinar at the end of this series.

Get DevOps Digest Sent to Your Inbox

You don’t need to remember to check back with us each week. Instead, get the digest delivered directly to your inbox. Subscribe to our 25-week DevOps Digest and we’ll get you where you need to go, one email at a time.

See Perforce Helix in Action!

Join us for a live demo every other Tuesday and see the best of Perforce Helix in 20 minutes. Save your spot!