May 28, 2009

Low disk-space checking for your server


Running out of disk space on your server is a situation best left to your disaster-recovery testing. There's no reason for significant data-loss if you've set up your server layout according to the best practices. As far as this situation is concerned it's mostly about putting the database and journal on separate disks. Still, an ounce of prevention is worth pound of cure (something that doesn't roll off the tongue nearly as well with metric units) and to that end, I wrote a little program to check for low-disk-space on your server.

It's intended to be run as a change-submittrigger so if disk-space is too low, it can prevent submissions. You might want to take a look at the following KB articles: triggers and daemons and debugging triggers. While only blocking submits doesn't guarantee that you won't run out of space (since the log, journal and db files can still grow), it will certainly get the attention of your users who will no doubt go running into your office en-masse.

The most straight-forward way to try this out is to run the program without any arguments and view its help message:

USAGE: space_check (-r) admin_info path1 size1 ... pathN sizeN
EXAMPLE: space_check [email protected]
/disk1/p4_db/ 500M
/disk2/p4_depot/ 5G
/disk3/p4_logs/ 100M

Available units of size are (M)egabytes and (G)igabytes.
Use the '-r' argument to report values while debugging.
Use an absolute path in the trigger definition for the best error reporting.
UNC paths and drive letters on Windows must have a trailing slash.

Then once you're satisfied with the results, install it as a trigger:

low_space_check change-submit //... "e:\p4_triggers\space_check.exe
[email protected] t:\p4_root\ 1G n:\p4_depot\ 5G m:\p4_journal\ 500M
t:\p4_logs\ 100M"

Now you can sit back and enjoy having another safeguard for your data-integrity. When disk space gets too low, your users will get a message like the following, sans coloring unless you add that in yourself.

*WARNING* ======================== *WARNING*
Minimum allowed disk space on "/" is 201G - currently at: 42

Submit *ABORTED* --- Please *CONTACT* the administrator at: [email protected]

Trigger program path: /disk4/p4_triggers/space_check
Depot ID: $Id: //depot/intranet/p4d/triggers/submit/ $

The trigger is a C++ program, so you need to compile it for your platform before you can run it. It should work on OSX, Linux, FreeBSD and Windows - or at least that's where I tried it out. If you see anything amiss, let me know!

Get it from the Public Depot here: free space checking trigger, and don't forget to check it into your depot with the "+k" filetype for keyword expansion.