October 31, 2014

Five ways to keep zombies, witches, ghosts and ghouls away from your Perforce installation

Traceability

In honor of Halloween, the Perforce Consultants have a few tips to help you keep zombies, witches, ghosts and ghouls away from your Perforce installation and to avoid things that go bump in the night! Here are a few treats that might make your life as a Perforce administrator easier.

ghouls dessert

Lay monitor ghosts to rest

We always recommend turning monitoring on so that all users can (via p4 monitor) see currently running commands on your server. Make sure your users know about it so that if the server is running slowly they can see the basic status.

Sometimes the monitor output can show commands that appear to have been running for many (hundreds even) hours - ghostly apparitions, or apparently zombie processes:

> p4 monitor show -l|ss -n idle
412 T buildsvr    216:20:41 login
3808 R buildsvr    00:27:06 sync source\...
4392 T fredb      259:21:49 login

In reality some of these processes are no longer active, and are usually the result of interrupted communication with the server. To be sure that the process is not active, you need to run something like “ps -elf | grep p4d" and check the process ids (PIDs) referenced and check that the PIDs shown in the monitor output are not actually running. On Windows you can use Process Monitor and check the task ids.

Clearing the entries if you are superuser is just about running “p4 monitor clear ” - the above example would be 412 or 4392.

diagnosing the entrails

Diagnosing the entrails - the gory guts of your server logs

It is always worth analyzing your server logs from time to time to keep an eye on the general health of your server. Make sure you have “p4 configure set server=3” turned on (or its equivalent for older servers). This means that your server log file contains start/stop times for all commands, with intermediate compute times as well for some commands (such as submit). 

These log files can be analyzed in various different ways:

For example, some analysis recently with a couple of customers highlighted issues such as:

  • Some users were using “default” client workspaces which mapped all depots visible to them – and commands within these workspaces were not performing well, and also affecting other users
  • Some build farms were badly configured, and were doing lots of unnecessary “sync –f” commands.
  • Sometimes you discover a new Jenkins server has been set up by an individual without you being notified!
  • Ancient perforce client versions in use from old automation scripts that had been rather forgotten about.

pumpkin zombie

Light your lanterns - keep the ghouls away!

Related to the suggestions above, there have been several presentations about Perforce server dashboards. These can be very beneficial, both for administrators but also for your general user population to that they can find information for themselves without having to send in unnecessary requests to support.

Examples include:

Frankenstein server creation - offline checkpoints and replica management made easy

With the open sourcing of the SDP (Server Deployment Package), there are now no excuses for not implementing offline checkpointing. Benefits include almost zero downtime checkpoints for DR, plus regular refresh of your metadata db.* files to help keep their size down and improve performance.

The standardized layout of the SDP file system structure also makes it easy to manage installations with multiple server instances. In addition being consistent about lots of different replicas across different machines makes that much simpler.

Don't get shut out - lockless reads to the rescue!

A recurring theme at the MERGE 2014 was the customer reports on the benefits of lockless reads being turned on. Michael Shield's presentation, Performance and Scalability Improvements in Perforce, has some useful highlights (slides 23-27 show results from customers EA and VMWare with dramatic before and after differences). This does require an upgrade of your Perforce server to 2013.3 or later with a rebuild of your metadata db.* files from a checkpoint. There is a useful SDP script (see above) which makes this upgrade painless and requires almost no downtime.

If you went to MERGE 2014 you know how good an event it was and how useful it was as both a networking and informational event. If you didn't, then the online presentations and slides will still make really useful reading - well worth taking some time to review.

Meanwhile, we hope you survive safely to All Saints Day on 1 November (Halloween = All Hallow's Eve)!

And we leave you with a final scary thought - you could be working with this mob (some Perforce UK staff plus a ringer)!

perforce staff halloween