Monitoring the Server
This chapter describes how you use p4d commands to monitor the server and its resources:
-
You use the p4 diskspace command to monitor diskspace usage.
-
You use the p4 monitor command to monitor processes.
-
You set server trace flags with the p4d startup command to diagnose problems.
-
You examine logs to track commands that exceed predetermined thresholds of resource usage.
-
You use the p4 monitor command to display information about locked files.
-
You use the
P4AUDIT
environment variable to enable the auditing of file access. -
You use a variety of logging commands to log information and manage log files.
Monitoring disk space usage
Use the p4 diskspace command to monitor diskspace usage. By default, p4 diskspace displays the amount of free space, diskspace used, and total capacity of any filesystem used by Perforce.
By default, the Perforce Server rejects commands when free space on the filesystems
housing the P4ROOT
, P4JOURNAL
, P4LOG
, or
TEMP
fall below 10 megabytes. To change this behavior, set the
filesys.P4ROOT.min
(and corresponding) configurables to your desired
limits:
Configurable |
Default Value |
Meaning |
---|---|---|
|
10M |
Minimum diskspace required on server root filesystem before server rejects commands. |
|
10M |
Minimum diskspace required on server journal filesystem before server rejects commands. |
|
10M |
Minimum diskspace required on server log filesystem before server rejects commands. |
|
10M |
Minimum diskspace required for temporary operations before server rejects commands. |
|
10M |
Minimum diskspace required for any depot before server rejects commands. (If
there is less than |
If the user account that runs the Perforce Server process is subject to disk quotas, the
Server observes these quotas with respect to the filesys.*.min
configurables, regardless of how much physical free space remains on the filesystem(s) in
question.
To estimate much disk space is currently occupied by specific files in a depot, use the p4 sizes command with a block size corresponding to that used by your storage solution. For example, the command:
p4 sizes -a -s -b 512 //depot/...
shows the sum (-s
) of all revisions (-a
) in
//depot/...
, as calculated with a block size of 512 bytes.
//depot/... 34161 files 277439099 bytes 5429111 blocks
The data reported by p4 sizes actually reflects the diskspace required when files are synced to a client workspace, but can provide a useful estimate of server-side diskspace consumption.
Monitoring processes
Use the p4 monitor command to observe and control Perforce-related processes running on your Perforce server machine.
The following sections explain how you enable process monitoring and list running processes,.
Enabling process monitoring
Server process monitoring requires minimal system resources, but you must enable
process monitoring for p4 monitor to work. To monitor all active
commands, set the monitor
configurable as follows:
p4 configure set monitor=1
Additional settings offer more options:
-
0
: Server process monitoring off. (Default) -
2
: monitor both active commands and idle connections. -
5
: monitor both active commands and idle connections, including a list of the files locked by the command for more than one second. -
10
: monitor both active commands and idle connections, including a list of the files locked by the command for more than one second, with lock wait times included in the lock information. -
25
: monitor both active commands and idle connections, including a list of the files locked by the command for any duration, with lock wait times included in the lock information.
How you set up monitoring levels 5, 10, and 25, depends on the platform where the server is running. See the description of the p4 monitor command in P4 Command Reference for more information.
Enabling idle processes monitoring
By default, IDLE
processes (often associated with custom
applications based on the Perforce API) are not included in the output of p4
monitor. To include idle processes in the default output of p4
monitor, use monitoring level 2.
p4 configure set monitor=2
To display idle processes, use the command:
p4 monitor show -s I
Listing running processes
To list the processes monitored by the Perforce server, use the command:
p4 monitor show
To restrict the display to processes currently in the running state, use the command:
p4 monitor show -s R
By default, each line of p4 monitor output looks like this:
pid
status
owner
hh:mm:ss
command
[args
]
where pid
is the UNIX process ID (or Windows thread ID),
status
is R
or T
depending on
whether the process is running or marked for termination, owner
is the Perforce user name of the user who invoked the command,
hh:mm:ss
is the time elapsed since the command was called, and
command
and args
are the command and
arguments as received by the Perforce server. For example:
$ p4 monitor show 74612 R qatool 00:00:47 job 78143 R edk 00:00:01 filelog 78207 R p4admin 00:00:00 monitor
To show the arguments with which the command was called, use the -a
(arguments) flag:
$ p4 monitor show -a 74612 R qatool 00:00:48 job job004836 78143 R edk 00:00:02 filelog //depot/main/src/proj/file1.c //dep 78208 R p4admin 00:00:00 monitor show -a
To obtain more information about user environment, use the -e
flag.
The -e
flag produces output of the form:
pid client IP-address status owner workspace hh:mm:ss
command
[args
]
where client
is the Perforce application (and version string
or API protocol level), IP-address
is the IP address of the
user's Perforce application, and workspace
is the name of the
calling user's current client workspace setting. For example:
$ p4 monitor show -e 74612 p4/2011.1 192.168.10.2 R qatool buildenvir 00:00:47 job 78143 192.168.10.4 R edk eds_elm 00:00:01 filelog 78207 p4/2011.1 192.168.10.10 R p4admin p4server 00:00:00 monitor
By default, all user names and (if applicable) client workspace names are truncated at
10 characters, and lines are truncated at 80 characters. To disable truncation, use the
-l
(long-form) option:
$ p4 monitor show -a -l 74612 R qatool 00:00:50 job job004836 78143 R edk 00:00:04 filelog //depot/main/src/proj/file1.c //dep ot/main/src/proj/file1.mpg 78209 R p4admin 00:00:00 monitor show -a -l
Only Perforce administrators and superusers can use the -a
,
-l
, and -e
options.
Setting server trace and tracking flags
To modify the behavior of command tracing or performance tracking, specify the
appropriate -v
flag to the
p4d startup command. Use subsystem
=value
P4LOG
or the -L
flag to specify the log file. For example:
logfile
p4d -r /usr/perforce -v server=2 -p 1666 -L /usr/perforce/logfile
Before you activate logging, make sure that you have adequate disk space.
Windows
When running Perforce as a Windows service, use the p4 set command
to set P4DEBUG
as a registry variable. You can also set these trace flags
when running p4d.exe as a server process from the command line.
Setting server debug levels on a Perforce server (p4d) has no effect on the debug level of a Perforce Proxy (p4p) process, and vice versa.
Higher levels of the Perforce server command tracing and tracking flags are typically recommended only for system administrators working with Perforce Technical Support to diagnose or investigate problems.
Command tracing
The server command trace flags and their meanings are as follows.
Trace flag |
Meaning |
---|---|
|
Disable server command logging. |
|
Logs server commands to the server log file. (As of release 2011.1, this is the default setting) |
|
In addition to data logged at level 1, logs server command completion and
basic information on CPU time used. Time elapsed is reported in seconds. On UNIX,
CPU usage (system and user time) is reported in milliseconds, as per
|
|
In addition to data logged at level 2, adds usage information for compute phases of p4 sync and p4 flush (p4 sync -k) commands. |
For command tracing, output appears in the specified log file, showing the date, time, username, IP address, and command for each request processed by the server.
Performance tracking
The Perforce Server produces diagnostic output in the server log whenever user commands
exceed certain predetermined thresholds of resource usage. Performance tracking is enabled
by default, and if P4DEBUG
is unset (or the tracking flag is not specified on
the command line), the tracking level is computed based on the number of users in the
license file.
Tracking flag |
Meaning |
---|---|
|
Turn off tracking. |
|
Track all commands. |
|
Track excess usage for a server with less than 10 users. |
|
Track excess usage for a server with less than 100 users. |
|
Track excess usage for a server with less than 1000 users. |
|
Track excess usage for a server with more than 1000 users. |
The precise format of the tracking output is undocumented and subject to change.
Showing information about locked files
You can use the -L
option of the p4 monitor to show
information about locked files. The information is collected only for the duration of the
p4 monitor command and is not persisted. See the description of the
p4 monitor command for more information about how to set up this kind of
monitoring.
The following sample output to the p4 monitor show -L command, shows the information displayed about locked files:
8764 R user 00:00:00 edit [server.locks/clients/88,d/ws4(W),db.locks(R),db.rev(R)] 8766 R user 00:00:00 edit [server.locks/clients/89,d/ws5(W),db.locks(R),db.rev(R)] 8768 R user 00:00:00 monitor
Following pid, status, owner, and time information, output shows two edit commands that
have various files locked, including the client workspace lock in exclusive mode for the
workspaces ws4
and ws5
, and
db.locks
and db.rev
tables in read-only mode.
Auditing user file access
The Perforce Server is capable of logging individual file accesses to an audit logfile.
Auditing is disabled by default, and is only enabled if P4AUDIT
is set to point
to the location of the audit log file, or if the server is started with the -A
flag.
auditlog
When auditing is enabled, the server adds a line to the audit log file every time file content is transferred from the server to the client. On an active server, the audit log file will grow very quickly.
Lines in the audit log appear in the form:
date
time
user
@client
clientIP
command
file
#rev
For example:
$ tail -2 auditlog 2011/05/09 09:52:45 karl@nail 192.168.0.12 diff //depot/src/x.c#1 2011/05/09 09:54:13 jim@stone 127.0.0.1 sync //depot/inc/file.h#1
If a command is run on the machine that runs the Perforce Server, the
clientIP
is shown as 127.0.0.1
.
If you are auditing server activity in a replicated environment, each of your build farm
or forwarding replica servers must have its own P4AUDIT
log set.
Logging and structured log files
The Perforce Server can be configured to write log files in a structured
(.csv
) format. Structured log files contain more detail than
conventional log files, are easier to parse, and the Perforce Server offers additional
commands to help customize your site's logging configuration. This section summarizes the
commands you use to manage logging, describes the use of structured logs, and explains how log
files are rotated.
Note
All p4d error and info logs are in UTF8 for a server in unicode mode. You need an UTF8 console or editor to properly render this log information.
Logging commands
You can use the following commands to work with logs.
Command |
Meaning |
---|---|
p4 logappend |
If the user log is enabled, write an entry to |
p4 logparse |
Parse a structured log file and return the logged data in tagged format |
p4 logrotate |
Rotate a named logfile, or, if no name is specified, all server logs. This
command applies only to structured logs; it does not rotate the unstructured
|
p4 logstat |
Report the file size of the journal ( |
p4 logtail |
Output the last block of the error log ( |
p4 logschema |
Return a description of the specified log record type. |
Enabling structured logging
To enable structured logging, set the
serverlog.file.
configurable to the name
of the file. Valid names for structured log files and the information logged are shown in
the following table.
N
Filename |
Description |
---|---|
|
All loggable events (commands, errors, audit, etc...) |
|
Audit events (audit, purge) |
|
The results of p4 login attempts. If the login failed, the reason for this is included in the log. Additional information provided by the authentication method is also included. |
|
Command events (command start, compute, and end) |
|
Error events (errors-failed, errors-fatal) |
|
Server events (startup, shutdown, checkpoint, journal rotation, etc.) |
|
Major events that occur during replica integrity checking. |
|
Command tracking (track-usage, track-rpc, track-db) |
|
User events; one record every time a user runs p4 logappend. |
Files do not have to be set in consecutive order; for example, this is fine:
p4 configure set serverlog.file.1=audit.csv p4 configure set serverlog.file.2=auth.csv p4 configure set serverlog.file.4=track.csv
Note
Enabling all structured logging files can consume considerable diskspace. See Structured logfile rotation for information on how to manage the size of the log file and the number of log rotations.
The value you specify for N
may not exceed 500.
Structured logfile rotation
Each of the configured serverlog.file.
files has its own corresponding
N
serverlog.maxmb.
and
N
serverlog.retain.
configurables. For each
configured server log type, these configurables control the maximum size (in megabytes) of
the logfile before rotation, and the number of rotated server logs retained by the server.
N
Structured log files are automatically rotated on checkpoint, journal creation,
overflow of associated serverlog.maxmb.
limit
(if configured), and the p4 logrotate command. You can disable log
rotation after journal rotation by setting the configurable
N
dmrotatewithinjnl
to 0
. Disabling this behavior can
help when you're doing frequent journal rotations and you want the log rotated on a
different schedule.
You can use the serverlog.counter.
configurable to create a counter that tracks the number of times a structured log file has
been rotated. For example, the following command creates a rotation counter called
N
myErrorsCount
:
p4 configure set serverlog.counter.3=myErrorsCount
Each time the errors.csv
log file is rotated, the counter is
increased by one. In addition, the name of the log file is changed to specify the
pre-incremented counter value. That is, if the counter myErrors
is 7, the
errors.csv
file is named errors-6.csv
.
You can create a counter for each file described in the preceding table. Do not use
system reserved counter names for your counter: change
,
maxCommitChange
, job
, journal
,
traits
, upgrade
.
The p4 logtail command returns the current value of the counter when you logtail that log. It also returns the current size of the log at the end of the output (along with the ending offset in the log). The size and offset are identical if p4 logtail reads to the end of the log. Security monitoring tools can use counters and the p4 logtail command in the process of scanning log files to monitor suspicious activity.