August 12, 2015

Helix DVCS – Clone Like a Pro, Part 2

Media & Entertainment
Healthcare
Traceability

You might have seen my previous post on how to initialize your personal server. I promised to continue the series with a post on the nuts and bolts of cloning, and here it is.

When we initialize a personal server with ‘p4 init,’ we want to version a local set of files not yet versioned. When we use ‘p4 clone’, we want to continue to work on a project that is already versioned somewhere else.

As you probably expect, a personal server cloned from an existing Helix server will automatically inherit the case sensitivity and Unicode settings of that server.

Most Helix Versioning Engines (P4Ds) store a large number of projects. It usually does not make sense to clone all of them into a single personal server, so we’ll need to identify which set of files we want to clone, using one of two ways: file path or remotes.

File path

Let’s say I want to clone P4PythonLib from the workshop. I can do this with the command

  p4 -d p4pythonlib -u sven_erik_knop clone \
      -p workshop.perforce.com:1666 \
      -f //guest/sven_erik_knop/P4Pythonlib/scripts/...

The user and directory options work the same way as for ‘p4 init’. The file path instructs ‘p4 clone’ to take all files below the specified directory with all their history.

Additionally, this command creates a local remote entry ‘origin’ used to push changes to and fetch changes from the source server.

Remotes

A remote (or remote spec) defines how depot files are mapped from local server to a remote server. On a local server, the remote spec also provides the address of the remote server. Let’s have a look at the spec the clone command created for us.

 

       $ cd p4pythonlib
       $ p4 remote origin
       RemoteID:   origin

       Address:    workshop.perforce.com:1666

       Owner:  sven_erik_knop

       Options:    unlocked nocompress

       Update: 2015/07/10 16:06:38

       Access: 2015/07/10 16:06:43

       Description:
           auto-generated from clone command

       LastFetch:  8554

       LastPush:   8631

       DepotMap:
           //stream/main/... //guest/sven_erik_knop/P4Pythonlib/scripts/...

The interesting parts here are the RemoteID, Owner, Address and DepotMap (I'll explain LastFetch and LastPush in another post).

Each remote has a unique name for identification and is owned by a single user. The address specifies the location of the remote server; this will be interesting when I write about push and fetch in one of the next blog posts.

The DepotMap field allows us to map one or several locations in the remote server depot to our local server. The mapping applies the same rule as a branch spec -- later mappings override previous mappings, and you can use exclusions to remove files you do not want to clone or fetch.

This allows us to clone a project into our personal server tailored to our needs, for example, pulling in all the resources required to build and test a project locally.

Note that the left-hand side of the DepotMap is “//stream/main/…”. I’ll explain the significance below.

Cloning via remote spec

If my whole team is cloning the same project, or if the mapping I require goes beyond a single file path, I can create a remote spec on the remote server. I have created such a remote spec on the workshop with the name “sknop.python”.

Such a remote spec acts as a template. The Address field is ignored and usually set to “localhost:1666”. The remote spec can be used by anyone to clone their personal server. During that cloning a local remote spec is created from the template and renamed to “origin”.

The full syntax is then

 

  p4 –d p4pythonlib –u sven_erik_knop clone \
       –p workshop.perforce.com:1666 \
       -r sknop.python

Effects of cloning

What does cloning actually do?

First of all, it creates a personal server in the same way that ‘p4 init’ does. Then it fetches all files and changes from the mapped files in the remote spec into the new server.

Without shallow cloning (see below) every single revision of each file in the path will be fetched, providing you with the full history of the cloned files. This is also true for deleted or moved files to preserve consistency when pushing changes back later.

So what about integrations?

Cloning Integrations

If both source and target of the integration are within the boundary of the remote mapping, the complete integration history will be cloned. Integrations from outside the boundary are ignored during cloning, although the action on the cloned revision is preserved (that is, branch or integrate instead of add or edit). This will give you a hint on the origin of the change.

A fun exercise is to broaden the mapping after the initial clone to include the source of the integration. Next time you run ‘p4 fetch’, the missing integration records will be transferred from the source server, completing your picture in your cloned server.

Local streams

You might have noticed that the cloned server created a streams depot called “stream” and a single stream called “//stream/main. This is independent of the source you cloned from, even if the source revisions live in a non-stream depot. The fact that your personal server uses streams has no immediate impact on your work, but it makes creating and switching branches easier, as we will explore in a future post.

If for some reason you do not like the names of the depot or the stream, you can create a remote spec in the source server and specify the target depot and stream in their, for example

  DepotMap:
        //my/trunk/...    //depot/source/...

When cloning using this remote, a streams depot called “my” with a stream called “//my/trunk” will be created.

Shallow cloning

An interesting option for the “p4 clone” command is shallow cloning, invoked with the option “-m depth”. This option will ensure that at most depth revisions are cloned and transferred, typically making the cloning process faster and your resulting personal server smaller.

Note that in most cases you still will receive multiple changes to create a consistent history of the file revisions you cloned. The change content is filtered to only show the files you mapped in the remote and the revisions transferred.

There are no limitations on a server cloned with the shallow option with respect to fetching and pushing changes, making the shallow clone a useful option for projects with deep history and large binaries.

Conclusion

Now that you know how to clone a project in your Helix Versioning Engine, you can start to explore how to use DVCS in your local projects. In the next post we will explore how your Perforce Helix administrator can enable you to clone from your central Helix versioning engine.

As usual, you can provide feedback via email or my twitter handle @p4sven.

Happy hacking!