August 15, 2011

Merging bug fixes, not files

Integration
Branching
Traceability

The idea of merging bugs has immediate appeal to release managers. On a daily basis, developers may think in terms of modifying files and submitting a batch of files (a changelist). But release managers usually think in terms of bugs or tasks slated for a release.

Perforce, via the job system, offers excellent support for working with bugs and tasks. The Defect Tracking Gateway and other integration mechanisms provide the tools to link jobs with external task management systems like Quality Center. (If you’re still typing bug numbers into check-in comments, you need to look at job-based defect tracking integration.)

When it comes time to perform codeline operations – merging changes between branches – the standard integration commands are very file oriented. Even the interchanges command reports on changelists that need to be merged, although a changelist may only offer part of a bug fix. There’s no single command to see which jobs (bugs or tasks) need to be merged from one branch to another.

Prompted by a conversation I had at the recent User Conference, I realized that it’d be really easy to write a script that offers job-based merging, at least at the proof-of-concept level. I had written a release manager’s dashboard a while back for P4V using the JavaScript API, so I decided to extend that.

Pending merges
Pending merges (changelists and jobs)

As you can see in the partial screenshot above, I'm using the interchanges command to list pending changelists that need to be merged for each branch spec listed in the applet. For each changelist, I'm running the fixes command to see what jobs are linked to that changelist. If you click on a job ID, the applet will find all the changelists fixed by that job, and attempt to selectively integrate each one. The files are left open in the default changelist for the user to review, resolve, and submit.

Merge by job
Merge by job

Is this applet perfect? Far from it. There are a lot of grey areas and inefficiencies to consider:

  • The interchanges command has significant performance implications on the server. Running this command repeatedly for a large set of branches may have a performance impact.
  • Some of the changelists for a job may have already been integrated. Integrating them again won’t do any damage, but it is inefficient.
  • Some changelists may already be partially integrated. Again, trying to integrate the changelist again won’t hurt anything, but it’s a bit inefficient.
  • Selective integration isn’t a habit I’m fond of, as it can complicate future integrations. (The new integration engine coming in the 2011.1 release handles these cases better than in the past.)
  • Most seriously, integrating changes out of order may introduce problems if the change I’m integrating depends on a previous change that I’ve skipped. There’s no reliable way to predict this problem. You could see if the same files were included in previous changes, but a change in one file may depend in subtle ways on a change in another file. Bottom line: if you merge by bug or task, you need to understand the file content well enough to anticipate this kind of problem.
  • Ideally, the applet would let you drive the process through to submit, selecting a changelist to use and invoking the resolve dialog.

Still, this applet feels ‘good enough’ that I’d probably try using it for a while.

Fair warning – you’ll need the 2011.1 version of P4V to take full advantage of this applet. (One of the reasons Perforce is a great place to work is that I get my hands on beta releases much earlier than I used to.) Keep an eye out for the public beta later this summer.

Users of our P4Eclipse plugin for Eclipse may recall that MergeQuest offers some support for viewing pending job merges. The source code for MergeQuest will be available in the future, so you could add extensions to MergeQuest if you wanted to drive the integration process by jobs as well.

So if this is so easy, why isn’t there just a Perforce command that does it? The answer lies in all of those grey areas. No version control tool can figure out if cherry-picking a change will cause dependency issues, and at Perforce, we readily admit when a problem requires human expertise to solve.

That being said, if you think we could provide better support for this form of task-based codeline management, be sure to let us know. As the unveiling of streams and P4Sandbox shows, we learn a lot from listening to you – so talk to us! Anyone want to be the first to file an enhancement request to add job-based merging to the upcoming stream graph?

And in the meantime, if you think you can improve my applet, please do. If you think you’ve made an improvement, I’ll be happy to incorporate your work and write about it. There may even be some Perforce swag involved…

You can find the source files and installation instructions in the public depot. You may also want to look at another script by Sven Knop. Sven's script lists jobs yet to be merged between two depot paths, but is written in Python, so it can be used from the command line or as a P4V custom tool.