Using Git as an API for Perforce - Part 1
I recently attended MERGE 2013, and happened to attend Matt Attaway’s walk-through of Perforce programming interfaces, entitled Perforce the Plentiful Platform. As Matt quickly established, Perforce provides an incredibly rich set of APIs for implementing build, test, or other automation processes on top of your Perforce source management system.
However, I was most intrigued by Matt’s assertion that we now have Git as a API for Perforce.
Since I was working on a problem where I needed to automate submits to the Perforce server, I thought, hmm, wonder if I could use Git?
In this part, I’m going to walk you through how to set up a Git Fusion server, and then use it to conﬁgure a special ecdump account that we will use later for automated submits.
In Part II., I will discuss how to set up the Git repositories and development environment for accomplishing the ﬁnal automation.
My Development Environment
I do most of my development on a Mac workstation, and use VMware Fusion to run the occasional VM. So I thought, maybe I should ﬁrst try the Git Fusion OVA - free to download here. That way I could experiment with an already conﬁgured system, rather than rolling my own. This turned out to be a time-saving decision.
In addition to the combined Git Fusion/P4D from the OVA install, I used my workstation to run the Perforce Visual Tools (P4V) and for running the normal p4 command line tools, and of course for running Git itself.
Incredibly, you can do 100% of the conﬁguration you need from a remote client, even though it is instructive to observe the outcomes of your conﬁguration experiments on the server from time to time.
Installing the OVA
Installing the OVA under VMware Fusion is pretty easy, you just drop the OVA on the VMware Fusion icon and you’re up and running. The VM comes up and asks you to conﬁgure the root, Git, and Perforce account passwords. Go ahead and do this. After that, you will need to ﬁgure out the IP address of the OVA and verify that you can connect from a remote terminal using the Linux accounts you just set up.
There is a nice admin screen on port 5480 that allows you to do some initial conﬁguration, which primarily means uploading ssh keys for existing or soon to be existing Perforce accounts. This is not a bad way to go to get started quickly, and so this is what I did. Later, I learned it is just as easy from a remote p4 client.
Exploring the OVA
I named my OVA gitconfusion because I was pretty much confused at this point about how the whole thing worked. I’m sticking with that name for the duration of this blog, but hopefully, after reading it, you’ll be less confused than I was.
Backing Up the OVA Configuration
Before you go too far, I recommend that you back up the initial P4D and Git Fusion conﬁgurations. This will enable you to easily reset P4D back to an initial state, at any time during your development cycle.
One way to achieve this would be to just snapshot the VM, and restore it as needed.
However, I wanted a faster process, and so I ended up using TAR:
# service p4d stop # cd / # tar czf p4gf_backup.tgz ./p4db ./home/git # service p4d start
To restore, just reverse the process after moving the old directories aside:
# service p4d stop # cd / # mkdir trash # mv ./p4db ./home/git trash # tar xf p4gf_backup.tgz # service p4d start # rm –rf trash
That last step is optional – you may want to keep the old directories around for a while, for reference or comparison purposes.
Using P4V to Connect to the OVA
To connect to the Perforce server via P4V or a the p4 command line client, use gitconfusion:1666, user admin.
You can also login to p4 as git-fusion-user, or super.
No passwords required.
One of the ﬁrst things I did after installing, was to ssh to the ova and examine the default accounts.
The perforce Linux Account
We are provided with a perforce LINUX account, which is handy for running P4 on the OVA host:
# id perforce uid=1001(perforce) gid=1002(perforce) groups=1002(perforce)
The git Linux Account
The git account is used for all Git Fusion operations on the OVA. Login as root to explore:
# id git uid=1002(git) gid=1003(git) groups=1003(git) # su - git git@localhost:~$ pwd /home/git git@localhost:~$ ls -a .bash_history .bash_logout .bashrc .git-fusion .git-fusion-profile .profile git@localhost:~$ env | grep P4 P4USER=git-fusion-user P4PORT=localhost:1666 git@localhost:~$ grep P4 .* .git-fusion-profile:declare -x P4PORT=localhost:1666 .git-fusion-profile:declare -x P4USER=git-fusion-user
Note that .git-fusion-profile is sourced from .bashrc
So far, no .ssh directory.
The git user crontab
In the OVA, a crontab is set up for the git user-id:
git@localhost:~$ crontab -l … * * * * bash -i -c p4gf_auth_update_authorized_keys.py
This entry is used to automatically update ~git/.ssh/authorized_keys whenever you add an ssh key ﬁle in the magic Perforce directory, which is:
Using the Admin Screen to Upload SSH Keys:
An alternative method to add ssh keys for a Perforce user is to use the management console on port 5480, which will present you with a dialog to upload an ssh public key ﬁle.
After you upload the ﬁle, you may still have to add a Perforce user account to match. You can do this in the OVA or from any command line client conﬁgured to talk to p4d on the OVA. Either way, use the p4 user command to get it done, for example:
$ p4 -u super user -f bob
How is SSH used to Map Users on the Git Fusion Server?
In ~git/.ssh/authorized_keys, the user's ssh signature is used to identify Git Fusion clients, and then map them to a Perforce user account. Please read the previous sentence again, for it is very easy to forget.
For example, here is one of my entries:
command="p4gf_auth_server.py --user=rtremain -- keyfp=f6:28:30:89:86:80:ac:2d:c8:12:c9:1d:17:e4:b7:16\ $SSH_ORIGINAL_COMMAND",no-port-forwarding,no-X11- forwarding,no-agent-forwarding,no-pty ssh-rsa AAAAB3NzaC1y... [truncated for illustration]
Note the --user argument – this is the Perforce user account the ssh session identiﬁed by the key will be mapped to.
Important note: please do not edit ~git/.ssh/authorized_keys manually – let Git Fusion manange it for you!
To Summarize, Let's Add the ECDUMP User:
There are three steps:
- add the Perforce user: p4 -u super user -f ecdump
- generate keys: ssh-keygen -f ~/.ssh/id_rsa.ecdump (do not enter a passphrase).
- add the public ssh key ﬁle to a p4 client and submit to:
As mentioned earlier, the ~git crontab entry that calls p4gf_auth_update_authorized_keys.py, will detect the submit and add a new command to ~git/.ssh/authorized_keys:
# grep ecdump .ssh/authorized_keys command="p4gf_auth_server.py --user=ecdump --keyfp= ...
If you don't see the new account line in authorized_keys after a minute or so, then check that the cron job is conﬁgured in the ~git account, and browse the p4 repository to make sure you added the key ﬁle in the correct location.
Note that for step 3), you will need to create a p4 client for modifying user keys. For example, I created a client named gitfusion-meta:
Client: gitfusion-meta ... Root: /tmp/gitfusion-meta View: //.git-fusion/... //gitfusion-meta/...
Time for Git!
Now that we have some ssh keys installed, lets use one of the sample p4 workspaces to clone a Git repository on your workstation:
Add a conﬁguration to ~/.ssh/config for the new account, for example:
Host ecdump.gitconfusion Hostname gitconfusion User git Protocol 2 IdentityFile ~/.ssh/id_rsa.ecdump IdentitiesOnly yes
This is a keen trick enabling you to maintain multiple identities for interacting with Git Fusion or any other remote system. For example, you may have one ssh identity you use for your normal work, another for GitHub, and another for SOURCEFORGE. The trick is to name the conﬁguration entry for it’s intended purpose, and to name the associated IdentityFile correctly. It is okay to have multiple identities, even for the same host, as long as you specify the Hostname attribute in your entry.
Then you need to make sure you supply the correct email account to Git. You can conﬁgure the following variables to override your default settings for Git:
$ export GIT_AUTHOR_NAME="EC Dumper" $ export GIT_AUTHOR_EMAIL=build@Perforce.com $ export GIT_COMMITTER_NAME="EC Dumper" $ export GIT_COMMITTER_EMAIL=build@Perforce.com #to check: $ git var -l
Now you are ready to clone one of the test views:
$ git clone ecdump.gitconfusion:Misc
Note the use of ecdump.gitconfusion. This will pick up the identity conﬁguration from ~/.ssh/config that we added earlier, substituting in the correct host, user, and key.
Next, make sure that you can commit and push your changes up:
$ cd Misc $ date > somefile $ git add somefile $ git commit -m somefile [master 96316ee] somefile 1 file changed, 1 insertion(+) create mode 100644 somefile $ git push ... remote: Perforce: Submitting new Git objects to Perforce: 6 To email@example.com:Misc 101c6aa..96316ee master -> master
If you get a error on the push, check that the email you have conﬁgured in Git matches the p4 email address from p4 users.
If the push was successful, you should see a description like this when you browse to your new ﬁle using P4V:
Imported from Git Author: EC Dumper
1368055322 -0700 Committer: EC Dumper@perforce.com> 1368055322 -0700 sha1: 96316ee42b33a2e7f798123efd7d4d1f647833b2 push-state: firstname.lastname@example.org>
Email Address is Important!
Git Fusion will not let you push unless your Git email is identiﬁed with your p4 user account. (It will let you clone.)
You only need to set a $GIT_AUTHOR_EMAIL environment variable to override your default Git email address (the one you have conﬁgured in ~/.gitconfig). However, it is better to set all of the Git email variables, as we did above, to make your changelist descriptions more informative and accurate.
Set these environment variables before you clone a repository, so that the local Git working directory will be conﬁgured correctly from the get-go, so to speak.
If you have already committed to your local Git repository with the wrong email, and you want to push it to Git Fusion, then you will need to correct your email to enable the push. Either reclone from Git Fusion with the correct email setting, (easy) or edit the previous commits using a procedure like this one (not so easy).
Remember to check your conﬁguration with: git var -l
Alternatively, use p4 to edit //.git-fusion/users/p4gf_usermap to map multiple emails to the same Perforce account. This could be a reasonable way to conﬁgure multiple automation jobs you want to map to the same Perforce account, regardless of the email address of the machine or process that is doing the submitting.
Remember, the p4gf_usermap mapping is not necessary if you conﬁgure your p4 user account with the same email you want to use with Git.
I hope that this tutorial has allowed you to quickly set up a Git Fusion instance for experimentation or development. You should now have a Git Fusion VM set up, and a back-up of the intial Git Fusion and P4D conﬁguration so you can quickly revert to a base state. You should have one or more sets of ssh keys established for identifying your development accounts to the server. You should be able to set up a sample Perforce client, clone it to Git, modify it using Git, and submit the changes back to Perforce via the Git Fusion agent.
Next time, I will walk you through how I set up the actual automation scripts we now have in place for submitting incremental source changes to our Perforce server, from our Electric Commander build automation system.
Finally, I hope you can name your Git Fusion server gf-savvy, to reﬂect your newfound knowledge!