August 29, 2011

Megalock: Detecting checkouts across streams

Integration
Streams
Traceability

Some files just can't or shouldn't be merged, at least not all the time. Binary files like office documents or graphics are an obvious example. Sometimes a software interface file may be tricky to merge when major updates are made.

Perforce supplies the ability to work in an exclusive checkout mode using the +l file type modifier, or files can be temporarily locked, preventing other checkins, using the p4 lock command. That's very handy, and also one of the reasons why Perforce normally wants you to indicate which files you're working on in advance.

But Perforce's exclusive checkout or locking tools don't work across branches. That is a potential problem, and one that some users have solved with scripts. The scripts always had to assume some knowledge of the branching model, in order to figure out where else to look. In other words, if you're working on a development branch and you want to make sure that a file is not being modified on the integration branch, you need to know where that integration branch lives in the depot. So these scripts always had to be heavily tailored to each environment.

Now that we have streams, the problem is a bit easier to manage. Perforce streams include information about the parent-child relationships, so the stream hierarchy can be constructed programmatically without making too many assumptions. I've written a script that I hope will serve as a starting point for cross-stream file locking.

The script is optimistically called Megalock with that goal in mind, but right now it's just a reporting tool. Given a single file in the current stream as defined by the active workspace, it traces that file through all sibling streams, and reports if the file is opened in other streams.

As an example, let's say that I intend to edit a database schema definition, schema.sql. I'm working in a development stream called //pb/dev-db-linux. I can use this script to see if anyone else is modifying the schema in another stream:

> python megalock.py db\schema.sql

Working in stream   //pb/dev-db-linux
Working on file     //pb/dev-db-linux/db/schema.sql

STREAM                        OPENED WHERE                  FILE
//pb/dev-db-linux             bruno@bruno_pb                //pb/dev-db-linux/db/schema.sql
//pb/dev-db                   not opened anywhere           //pb/dev-db/db/schema.sql
//pb/2.0-int                  not opened anywhere           //pb/2.0-int/db/schema.sql
//pb/main                     not opened anywhere           //pb/main/db/schema.sql
//pb/1.5-r                    not opened anywhere           //pb/1.5-r/db/schema.sql
//pb/1.0-r                    not opened anywhere           //pb/1.0-r/db/schema.sql
//pb/1.5.1-p                  bruno@bruno_pb_3              //pb/1.5.1-p/db/schema_mod.sql
//pb/dev1.0                   not opened anywhere           //pb/dev1.0/db/schema.sql
//pb/dev-gui                  no mapping//pb/dev-db-linux-child       bruno@bruno_pb_2              //pb/dev-db-linux-child/db/schema.sql
//pb/sb-1                     not opened anywhere           //pb/sb-1/db/schema.sql
//pb/sb-2                     not opened anywhere           //pb/sb-2/db/schema.sql

The script lets me know that the schema is also being modified in a patch maintenance stream (//pb/1.5.1-p) and in another development stream (//pb/dev-db-linux-child). What I do now is up to me. I'd probably at least contact the other users and see what they're up to. Maybe I'll hold off on making my changes until they're done, or maybe their changes are minor and I'll continue working.

The script takes advantage of the fact that the stream view also accounts for modular development practices and other complications. Looking at my example output again, notice that there is no mapping reported for the //pb/dev-gui stream. That stream is for GUI work, and just imports the database assets from other streams. If anyone has the database schema file checked out locally in that stream or any of its children, I don't care, because that change will never affect me.

The script does some checking to see if the file in question has been renamed in other streams. You'll notice that the file is checked out in the //pb/1.5.1-p stream, but it was renamed to schema_mod.sql. I won't guarantee that the script handles all refactorings correctly, but it handles the simple cases for now.

Let's look at the stream graph to get a sense of how valuable this information is. In the picture below the streams where the file is being modified are circled in blue, while the streams where this file isn't relevant are circled in red.

Checkouts in sibling streams
Checkouts in sibling streams

It would be difficult for me to manually track down this information. Even as a reporting tool, I think this script offers a lot of value. But of course there are a lot of ways to make it better in the future, to get closer to the goal of exclusive checkouts across streams.

Perforce doesn't offer triggers for operations that open a file, but this script could be used as the basis of a broker filter script. With some modifications, this script could provide a new pseudo-command, making the reporting features more easily available to all users. More ambitiously, the commands that open a file could be filtered through the broker. For certain types of files, these commands could be rejected if they are open in sibling streams. You'll need to decide if you want to go down the route of preventing checkouts rather than just providing information, and of course running this script on a large number of files would impose some extra load on the Perforce server.

Usage information, limitations, and other notes are in the script header. I'd be very interested to hear your feedback on this approach and whether it is useful for you.