July 2, 2013

Using Git as an API for Perforce - Part 1

Git at Scale
Traceability

git fusion icon

Introduction

I recently attended MERGE 2013, and happened to attend Matt Attaway’s walk-through of Perforce programming interfaces, entitled Perforce the Plentiful Platform. As Matt quickly established, Perforce provides an incredibly rich set of APIs for implementing build, test, or other automation processes on top of your Perforce source management system.

However, I was most intrigued by Matt’s assertion that we now have Git as a API for Perforce.

Since I was working on a problem where I needed to automate submits to the Perforce server, I thought, hmm, wonder if I could use Git?

In this part, I’m going to walk you through how to set up a Git Fusion server, and then use it to configure a special ecdump account that we will use later for automated submits.

In Part II., I will discuss how to set up the Git repositories and development environment for accomplishing the final automation.

My Development Environment

I do most of my development on a Mac workstation, and use VMware Fusion to run the occasional VM. So I thought, maybe I should first try the Git Fusion OVA - free to download here. That way I could experiment with an already configured system, rather than rolling my own. This turned out to be a time-saving decision.

In addition to the combined Git Fusion/P4D from the OVA install, I used my workstation to run the Perforce Visual Tools (P4V) and for running the normal p4 command line tools, and of course for running Git itself.

Incredibly, you can do 100% of the configuration you need from a remote client, even though it is instructive to observe the outcomes of your configuration experiments on the server from time to time.

Installing the OVA

Installing the OVA under VMware Fusion is pretty easy, you just drop the OVA on the VMware Fusion icon and you’re up and running. The VM comes up and asks you to configure the root, Git, and Perforce account passwords. Go ahead and do this. After that, you will need to figure out the IP address of the OVA and verify that you can connect from a remote terminal using the Linux accounts you just set up.

There is a nice admin screen on port 5480 that allows you to do some initial configuration, which primarily means uploading ssh keys for existing or soon to be existing Perforce accounts. This is not a bad way to go to get started quickly, and so this is what I did. Later, I learned it is just as easy from a remote p4 client.

Exploring the OVA

I named my OVA gitconfusion because I was pretty much confused at this point about how the whole thing worked. I’m sticking with that name for the duration of this blog, but hopefully, after reading it, you’ll be less confused than I was.

Backing Up the OVA Configuration

Before you go too far, I recommend that you back up the initial P4D and Git Fusion configurations. This will enable you to easily reset P4D back to an initial state, at any time during your development cycle.

One way to achieve this would be to just snapshot the VM, and restore it as needed.

However, I wanted a faster process, and so I ended up using TAR:

# service p4d stop
# cd /
# tar czf p4gf_backup.tgz ./p4db ./home/git
# service p4d start 

To restore, just reverse the process after moving the old directories aside:

 
# service p4d stop
# cd /
# mkdir trash
# mv ./p4db ./home/git trash
# tar xf p4gf_backup.tgz
# service p4d start
# rm –rf trash

That last step is optional – you may want to keep the old directories around for a while, for reference or comparison purposes.

Using P4V to Connect to the OVA

To connect to the Perforce server via P4V or a the p4 command line client, use gitconfusion:1666, user admin.

You can also login to p4 as git-fusion-user, or super.

No passwords required.

OVALINUX Accounts

One of the first things I did after installing, was to ssh to the ova and examine the default accounts.

The perforce Linux Account

We are provided with a perforce LINUX account, which is handy for running P4 on the OVA host:

# id perforce
uid=1001(perforce) gid=1002(perforce) groups=1002(perforce)

The git Linux Account

The git account is used for all Git Fusion operations on the OVA. Login as root to explore:

 # id git
uid=1002(git) gid=1003(git) groups=1003(git)
# su - git
git@localhost:~$ pwd
/home/git
git@localhost:~$ ls -a
.bash_history .bash_logout .bashrc
.git-fusion .git-fusion-profile .profile
git@localhost:~$ env | grep P4
P4USER=git-fusion-user
P4PORT=localhost:1666
git@localhost:~$ grep P4 .*
.git-fusion-profile:declare -x P4PORT=localhost:1666
.git-fusion-profile:declare -x P4USER=git-fusion-user

Note that .git-fusion-profile is sourced from .bashrc

So far, no .ssh directory.

The git user crontab

In the OVA, a crontab is set up for the git user-id:

git@localhost:~$ crontab -l
…
* * * * bash -i -c p4gf_auth_update_authorized_keys.py

This entry is used to automatically update ~git/.ssh/authorized_keys whenever you add an ssh key file in the magic Perforce directory, which is:

//.git-fusion/users/<name>/keys/<public_key_file>

Remember that!

Using the Admin Screen to Upload SSH Keys:

An alternative method to add ssh keys for a Perforce user is to use the management console on port 5480, which will present you with a dialog to upload an ssh public key file.

After you upload the file, you may still have to add a Perforce user account to match. You can do this in the OVA or from any command line client configured to talk to p4d on the OVA. Either way, use the p4 user command to get it done, for example:

$ p4 -u super user -f bob

How is SSH used to Map Users on the Git Fusion Server?

In ~git/.ssh/authorized_keys, the user's ssh signature is used to identify Git Fusion clients, and then map them to a Perforce user account. Please read the previous sentence again, for it is very easy to forget.

For example, here is one of my entries:

command="p4gf_auth_server.py --user=rtremain --
keyfp=f6:28:30:89:86:80:ac:2d:c8:12:c9:1d:17:e4:b7:16\
$SSH_ORIGINAL_COMMAND",no-port-forwarding,no-X11-
forwarding,no-agent-forwarding,no-pty ssh-rsa AAAAB3NzaC1y...
[truncated for illustration]

Note the --user argument – this is the Perforce user account the ssh session identified by the key will be mapped to.

Important note: please do not edit ~git/.ssh/authorized_keys manually – let Git Fusion manange it for you!

To Summarize, Let's Add the ECDUMP User:

There are three steps:

  1. add the Perforce user: p4 -u super user -f ecdump
  2. generate keys: ssh-keygen -f ~/.ssh/id_rsa.ecdump (do not enter a passphrase).
  3. add the public ssh key file to a p4 client and submit to:
//.git-fusion/users/ecdump/keys/id_rsa.pub

As mentioned earlier, the ~git crontab entry that calls p4gf_auth_update_authorized_keys.py, will detect the submit and add a new command to ~git/.ssh/authorized_keys:

# grep ecdump .ssh/authorized_keys
command="p4gf_auth_server.py --user=ecdump --keyfp= ...

If you don't see the new account line in authorized_keys after a minute or so, then check that the cron job is configured in the ~git account, and browse the p4 repository to make sure you added the key file in the correct location.

Note that for step 3), you will need to create a p4 client for modifying user keys. For example, I created a client named gitfusion-meta:

Client: gitfusion-meta
...
Root: /tmp/gitfusion-meta
View:
//.git-fusion/... //gitfusion-meta/...

Time for Git!

Now that we have some ssh keys installed, lets use one of the sample p4 workspaces to clone a Git repository on your workstation:

Add a configuration to ~/.ssh/config for the new account, for example:

Host ecdump.gitconfusion
  Hostname gitconfusion
  User git
  Protocol 2
  IdentityFile ~/.ssh/id_rsa.ecdump
  IdentitiesOnly yes

This is a keen trick enabling you to maintain multiple identities for interacting with Git Fusion or any other remote system. For example, you may have one ssh identity you use for your normal work, another for GitHub, and another for SOURCEFORGE. The trick is to name the configuration entry for it’s intended purpose, and to name the associated IdentityFile correctly. It is okay to have multiple identities, even for the same host, as long as you specify the Hostname attribute in your entry.

Then you need to make sure you supply the correct email account to Git. You can configure the following variables to override your default settings for Git:

$ export GIT_AUTHOR_NAME="EC Dumper"
$ export GIT_AUTHOR_EMAIL=build@Perforce.com
$ export GIT_COMMITTER_NAME="EC Dumper"
$ export GIT_COMMITTER_EMAIL=build@Perforce.com
#to check:
$ git var -l

Now you are ready to clone one of the test views:

$ git clone ecdump.gitconfusion:Misc

Note the use of ecdump.gitconfusion. This will pick up the identity configuration from ~/.ssh/config that we added earlier, substituting in the correct host, user, and key.

Next, make sure that you can commit and push your changes up:

$ cd Misc
$ date > somefile
$ git add somefile
$ git commit -m somefile
[master 96316ee] somefile
 1 file changed, 1 insertion(+)
 create mode 100644 somefile
$ git push
...
remote: Perforce: Submitting new Git objects to Perforce: 6
To git@ecdump.gitconfusion:Misc
 101c6aa..96316ee master -> master

If you get a error on the push, check that the email you have configured in Git matches the p4 email address from p4 users.

If the push was successful, you should see a description like this when you browse to your new file using P4V:

Imported from Git
 Author: EC Dumper  1368055322 -0700
 Committer: EC Dumper  1368055322 -0700
 sha1: 96316ee42b33a2e7f798123efd7d4d1f647833b2
 push-state: complete@perforce.com>@perforce.com>

Email Address is Important!

Git Fusion will not let you push unless your Git email is identified with your p4 user account. (It will let you clone.)

You only need to set a $GIT_AUTHOR_EMAIL environment variable to override your default Git email address (the one you have configured in ~/.gitconfig). However, it is better to set all of the Git email variables, as we did above, to make your changelist descriptions more informative and accurate.

Set these environment variables before you clone a repository, so that the local Git working directory will be configured correctly from the get-go, so to speak.

If you have already committed to your local Git repository with the wrong email, and you want to push it to Git Fusion, then you will need to correct your email to enable the push. Either reclone from Git Fusion with the correct email setting, (easy) or edit the previous commits using a procedure like this one (not so easy).

Remember to check your configuration with: git var -l

Alternatively, use p4 to edit //.git-fusion/users/p4gf_usermap to map multiple emails to the same Perforce account. This could be a reasonable way to configure multiple automation jobs you want to map to the same Perforce account, regardless of the email address of the machine or process that is doing the submitting.

Remember, the p4gf_usermap mapping is not necessary if you configure your p4 user account with the same email you want to use with Git.

Summary

I hope that this tutorial has allowed you to quickly set up a Git Fusion instance for experimentation or development. You should now have a Git Fusion VM set up, and a back-up of the intial Git Fusion and P4D configuration so you can quickly revert to a base state. You should have one or more sets of ssh keys established for identifying your development accounts to the server. You should be able to set up a sample Perforce client, clone it to Git, modify it using Git, and submit the changes back to Perforce via the Git Fusion agent.

Next time, I will walk you through how I set up the actual automation scripts we now have in place for submitting incremental source changes to our Perforce server, from our Electric Commander build automation system.

Finally, I hope you can name your Git Fusion server gf-savvy, to reflect your newfound knowledge!