Monitoring the Server
This chapter describes how you use
p4d commands to monitor the
server and its resources:
- You use the
p4 diskspacecommand to monitor diskspace usage.
- You use the
p4 monitorcommand to monitor processes.
- You set server trace flags with the
p4dstartup command to diagnose problems.
- You examine logs to track commands that exceed predetermined thresholds of resource usage.
- You use the
p4 monitorcommand to display information about locked files.
- You use the
P4AUDITenvironment variable to enable the auditing of file access.
- You use a variety of logging commands to log information and manage log files.
Monitoring disk space usage
p4 diskspace command to monitor diskspace usage. By
p4 diskspace displays the amount of free space,
diskspace used, and total capacity of any filesystem used by Perforce.
By default, the Perforce Server rejects commands when free space on the
filesystems housing the
below 10 megabytes. To change this behavior, set the
filesys.P4ROOT.min (and corresponding) configurables to your desired
Minimum diskspace required on server root filesystem before server rejects commands.
Minimum diskspace required on server journal filesystem before server rejects commands.
Minimum diskspace required on server log filesystem before server rejects commands.
Minimum diskspace required for temporary operations before server rejects commands.
Minimum diskspace required for any depot before server rejects
commands. (If there is less than
If the user account that runs the Perforce Server process is subject to
disk quotas, the Server observes these quotas with respect to the
filesys.*.min configurables, regardless of how
much physical free space remains on the filesystem(s) in question. The
next section explains the options you have in reconfiguring default
Specifying values for filesys configurables
filesys.*.min values, you have the option of specifying an
absolute number or a percentage indicating a portion of the current space.
So, there are six possible numeric formats you can use, as shown in the
A plain number, used as is.
A number in kilobytes
For example, the following command sets
$ p4 configure set filesys.P4TEMP.min=100K
A number in megabytes
For example, the following command sets
$ p4 configure set filesys.P4ROOT.min=10M
A number in gigabytes.
For example, the following command sets
$ p4 configure set filesys.P4JOURNAL.min=1G
A number in terabytes.
A number as a percentage of the current space.
For example, the following command means that at least ten percent of
the total disk space must be free and available for
$ p4 configure set filesys.P4ROOT.min=10%
Determining available disk space
To estimate how much disk space is currently occupied by specific files
in a depot, use the
p4 sizes command with a block size
corresponding to that used by your storage solution. For example, the
$ p4 sizes -a -s -b 512 //depot/...
shows the sum (
-s) of all revisions (
//depot/..., as calculated with a block size
of 512 bytes.
//depot/... 34161 files 277439099 bytes 5429111 blocks
The data reported by
p4 sizes actually reflects the diskspace
required when files are synced to a client workspace, but can provide a
useful estimate of server-side diskspace consumption.
p4 monitor command to observe and control
Perforce-related processes running on your Perforce server machine.
The following sections explain how you enable process monitoring and list running processes,.
Enabling process monitoring
Server process monitoring requires minimal system resources, but you
must enable process monitoring for
p4 monitor to work. To
monitor all active commands, set the
monitor configurable as follows:
$ p4 configure set monitor=1
Additional settings offer more options:
0: Server process monitoring off. (Default)
2: monitor both active commands and idle connections.
5: monitor both active commands and idle connections, including a list of the files locked by the command for more than one second.
10: monitor both active commands and idle connections, including a list of the files locked by the command for more than one second, with lock wait times included in the lock information.
25: monitor both active commands and idle connections, including a list of the files locked by the command for any duration, with lock wait times included in the lock information.
How you set up monitoring levels
25, depends on the
platform where the server is running. See the description of the
monitor command in P4 Command Reference for more information.
Enabling idle processes monitoring
IDLE processes (often associated with custom applications
based on the Perforce API) are not included in the output of
monitor. To include idle processes in the default output of
monitor, use monitoring level
$ p4 configure set monitor=2
To display idle processes, use the command:
$ p4 monitor show -s I
Listing running processes
To list the processes monitored by the Perforce server, use the command:
$ p4 monitor show
To restrict the display to processes currently in the running state, use the command:
$ p4 monitor show -s R
By default, each line of
p4 monitor output looks like this:
pid is the UNIX process ID (or Windows thread ID),
T depending on whether the process is running or marked for
owner is the Perforce user name of the user who invoked
hh:mm:ss is the time elapsed since the command was
args are the command and arguments as
received by the Perforce server. For example:
$ p4 monitor show 74612 R qatool 00:00:47 job 78143 R edk 00:00:01 filelog 78207 R p4admin 00:00:00 monitor
To show the arguments with which the command was called, use the
$ p4 monitor show -a 74612 R qatool 00:00:48 job job004836 78143 R edk 00:00:02 filelog //depot/main/src/proj/file1.c //dep 78208 R p4admin 00:00:00 monitor show -a
To obtain more information about user environment, use the
-e flag produces output of the form:
pid client IP-address status owner workspace hh:mm:ss command[
client is the Perforce application (and version string or API
IP-address is the IP address of the user’s Perforce
workspace is the name of the calling user’s current
client workspace setting. For example:
$ p4 monitor show -e 74612 p4/2011.1 192.168.10.2 R qatool buildenvir 00:00:47 job 78143 192.168.10.4 R edk eds_elm 00:00:01 filelog 78207 p4/2011.1 192.168.10.10 R p4admin p4server 00:00:00 monitor
By default, all user names and (if applicable) client workspace names
are truncated at 10 characters, and lines are truncated at 80
characters. To disable truncation, use the
-l (long-form) option:
$ p4 monitor show -a -l 74612 R qatool 00:00:50 job job004836 78143 R edk 00:00:04 filelog //depot/main/src/proj/file1.c //dep ot/main/src/proj/file1.mpg 78209 R p4admin 00:00:00 monitor show -a -l
Only Perforce administrators and superusers can use the
Setting server trace and tracking flags
To modify the behavior of command tracing or performance tracking,
specify the appropriate
flag to the
p4d startup command. Use
P4LOG or the
`logfile` flag to specify the log file. For example:
$ p4d -r /usr/perforce -v server=2 -p 1666 -L /usr/perforce/logfile
Before you activate logging, make sure that you have adequate disk space.
When running Perforce as a Windows service, use the
command to set
P4DEBUG as a registry variable. You can also set these
trace flags when running
p4d.exe as a server process from the
Setting server debug levels on a Perforce server (
p4d) has no
effect on the debug level of a Perforce Proxy (
p4p) process, and
Higher levels of the Perforce server command tracing and tracking flags are typically recommended only for system administrators working with Perforce Technical Support to diagnose or investigate problems.
The server command trace flags and their meanings are as follows.
Disable server command logging.
Logs server commands to the server log file.
(As of release 2011.1, this is the default setting.)
In addition to data logged at level 1, logs server command completion
and basic information on CPU time used. Time elapsed is reported in
seconds. On UNIX, CPU usage (system and user time) is reported in
milliseconds, as per
In addition to data logged at level 2, adds usage information for
compute phases of
For command tracing, output appears in the specified log file, showing the date, time, username, IP address, and command for each request processed by the server.
The Perforce Server produces diagnostic output in the server log
whenever user commands exceed certain predetermined thresholds of
resource usage. Performance tracking is enabled by default, and if
P4DEBUG is unset (or the tracking flag is not specified on the command
line), the tracking level is computed based on the number of users in
the license file.
Turn off tracking.
Track all commands.
Track excess usage for a server with less than 10 users.
Track excess usage for a server with less than 100 users.
Track excess usage for a server with less than 1000 users.
Track excess usage for a server with more than 1000 users.
The precise format of the tracking output is undocumented and subject to change.
Showing information about locked files
You can use the
-L option of the
p4 monitor to show
information about locked files. The information is collected only for
the duration of the
p4 monitor command and is not persisted. See
the description of the
p4 monitor command for more information
about how to set up this kind of monitoring.
The following sample output to the
p4 monitor show -L command,
shows the information displayed about locked files:
8764 R user 00:00:00 edit [server.locks/clients/88,d/ws4(W),db.locks(R),db.rev(R)] 8766 R user 00:00:00 edit [server.locks/clients/89,d/ws5(W),db.locks(R),db.rev(R)] 8768 R user 00:00:00 monitor
Following pid, status, owner, and time information, output shows two
edit commands that have various files locked, including the client
workspace lock in exclusive mode for the workspaces
db.rev tables in read-only mode.
Auditing user file access
The Perforce Server is capable of logging individual file accesses to an
audit logfile. Auditing is disabled by default, and is only enabled if
P4AUDIT is set to point to the location of the audit log file, or if
the server is started with the
-A auditlog flag.
When auditing is enabled, the server adds a line to the audit log file every time file content is transferred from the server to the client. On an active server, the audit log file will grow very quickly.
Lines in the audit log appear in the form:
$ tail -2 auditlog 2011/05/09 09:52:45 karl@nail 192.168.0.12 diff //depot/src/x.c#1 2011/05/09 09:54:13 jim@stone 127.0.0.1 sync //depot/inc/file.h#1
If a command is run on the machine that runs the Perforce Server, the
clientIP is shown as
If you are auditing server activity in a replicated environment, each of
your build farm or forwarding replica servers must have its own
P4AUDIT log set.
Logging and structured log files
The Perforce Server can be configured to write log files in a structured
.csv) format. Structured log files contain more detail than
conventional log files, are easier to parse, and the Perforce Server
offers additional commands to help customize your site’s logging
configuration. This section summarizes the commands you use to manage
logging, describes the use of structured logs, and explains how log
files are rotated.
p4d error and info logs are in UTF8 for a server in unicode
mode. You need an UTF8 console or editor to properly render this log
You can use the following commands to work with logs.
If the user log is enabled, write an entry to
Parse a structured log file and return the logged data in tagged format
Rotate a named logfile, or, if no name is specified, all server logs.
This command applies only to structured logs; it does not rotate the
Report the file size of the journal (
Output the last block of the error log (
Return a description of the specified log record type.
Enabling structured logging
To enable structured logging, set the
configurable to the name of the file. Valid names for structured log
files and the information logged are shown in the following table.
You can use a file path in conjunction with the file name.
You must use one of the file names specified in the table. If you use an arbitrary name, no data will be logged to the file you specify.
All loggable events (commands, errors, audit, etc…)
Audit events (audit, purge)
The results of
Command events (command start, compute, and end)
Error events (errors-failed, errors-fatal)
Server events (startup, shutdown, checkpoint, journal rotation, etc.)
Major events that occur during replica integrity checking.
Log the full network route of authenticated client connections. Errors
Command tracking (track-usage, track-rpc, track-db)
User events; one record every time a user runs
Files do not have to be set in consecutive order; for example, this is fine:
$ p4 configure set serverlog.file.1=audit.csv $ p4 configure set serverlog.file.2=auth.csv $ p4 configure set serverlog.file.4=track.csv
Enabling all structured logging files can consume considerable diskspace. See Structured logfile rotation for information on how to manage the size of the log file and the number of log rotations.
The value you specify for
N may not exceed 500.
Structured logfile rotation
Each of the configured
files has its own corresponding
configurables. For each configured server log type, these configurables
control the maximum size (in megabytes) of the logfile before rotation,
and the number of rotated server logs retained by the server.
Structured log files are automatically rotated on checkpoint, journal
creation, overflow of associated
limit (if configured), and the
p4 logrotate command. You can
disable log rotation after journal rotation by setting the configurable
0. Disabling this behavior can help when you’re
doing frequent journal rotations and you want the log rotated on a
You can use the
configurable to create a counter that tracks the number of times a
structured log file has been rotated. For example, the following command
creates a rotation counter called
$ p4 configure set serverlog.counter.3=myErrorsCount
Each time the
errors.csv log file is rotated, the counter is increased
by one. In addition, the name of the log file is changed to specify the
pre-incremented counter value. That is, if the counter
myErrors is 7,
errors.csv file is named
You can create a counter for each file described in the preceding table.
Do not use system reserved counter names for your counter:
p4 logtail command returns the current value of the counter
when you logtail that log. It also returns the current size of the log
at the end of the output (along with the ending offset in the log). The
size and offset are identical if
p4 logtail reads to the end of
the log. Security monitoring tools can use counters and the
logtail command in the process of scanning log files to monitor