October 19, 2010

Down for Maintenance - Perforce in Single User Mode

Got the Car Apart in the Garage

Perforce is the engine of progress in many organizations -- and sometimes that engine needs maintenance.  When Perforce is down for maintenance, it's not good enough to rely on emails telling users they're not supposed to use Perforce during your planned maintenance window.  Even if you could rely on all the humans to read and heed your communications, it's likely that various automated process also interact with Perforce, such as build systems or defect trackers.

Often maintenance operations with Perforce start by taking a checkpoint.  From just before the time you take that checkpoint, you want to be sure no one else is on the system, and no other processes are using Perforce.  That way, should something go horribly wrong in your maintenance operation and you need to rollback to the checkpoint, you can guarantee that no one's work was lost.

The answer is to run Perforce in single user mode.  For example, you might want Perforce to be available only to administrators during a maintenance window.  You can achieve the desired effect of single user mode by running Perforce a different port, other than the well-known port on which it usually runs.  For example, if your P4PORT value is usually perforce.myco.com:1742, you might document a maintenance procedure something like this:

  1. Communicate, communicate, communicate!  'Nuff said.
  2. Disable any 'crontab' or Scheduled Tasks (or whatever your process scheduler is) that might interfere with the maintenance.
  3. Run 'p4 admin stop' to stop the 'p4d' process safely.  Any writes to metadata that made it in are "safe" -- you can roll back to them after maintenance.
  4. Change P4PORT to a secret "maintenance mode" value known only to Perforce admins, perforce.myco.com:1743.
  5. Start the 'p4d' process in the usual way, ensuring that it is running on the "maintenance mode" port and specifying P4ROOT, P4JOURNAL, P4LOG, etc.
  6. Take a checkpoint to create the rollback point.
  7. Do whatever maintenance you had in mind to do.
  8. Do whatever sanity checks or other testing you had in mind to do (while the service is still unavailable to users).
  9. Run 'p4 admin stop' to stop the 'p4d' process safely, and then swap back to the original P4PORT, and restart 'p4d'.  Users will now see Perforce.
  10. Don't forget to re-enable your crontab entries/Scheduled Tasks.
  11. Communicate some more!

Flexible License Files

What about that license file that "locks" Perforce to a particular port on your server's IP address?  Hakuna Matata ("No Worries!")  The port-lock doesn't prevent you from starting Perforce on another port.  When you run 'p4d' against a port other than the one identified in the license file, 'p4 info' reports that it is running in restricted mode.  Restricted mode is not suitable for live production, but it's perfect for emulating for single user mode.

HA, DR, and Replicas

If you have High Availability, Disaster Recovery, and/or Read-Only Replicas in place, your maintenance window procedures will need to take into account the impact on these systems.