Perforce server version 2010.1 introduced support for the replication of metadata, but did not support the automated replication of versioned file data. Starting with the 2010.2 release, the Perforce Server now supports several new commands to simplify configuration and replication between Perforce servers of both metadata and versioned files.
Combined with a centralized authorization server (see Centralized authorization server), Perforce administrators can configure the Perforce Broker (see
Chapter 11, The Perforce Broker) to redirect commands to read-only replica servers to balance load efficiently across an arbitrary number of replica servers.
|
•
|
To use p4 pull, the master and replica servers must be revision 2010.2 or higher.
|
|
•
|
To use p4 replicate, the master and replica servers must be revision 2009.2 or higher.
|
|
•
|
p4 replicate and p4 pull (when replicating metadata) do not read compressed journals. Therefore, the master server must not compress rotated journals until the replica server has fetched all journal records from older journals.
|
|
|
|
|
|
|
|
|
|
|
|
Because p4 configure stores its data on the master server, all replica servers automatically pick up any changes you make.
|
|
|
When you use p4 configure on your master server, you can specify different sets of configurables for each named server. Each named server, upon startup, refers to its own set of configurables, and ignores configurables set for other servers.
|
|
|
A new type of user intended for authentication of server-to-server communications. Service users have extremely limited access to the depot and do not consume Perforce licenses.
To make logs easier to read, create one service user on your master server for each replica or proxy in your network of Perforce Servers.
|
|
|
In -M readonly mode, the Perforce Server denies any command that attempts to write to server metadata. In this mode, a command such as p4 sync (which updates the server's have list) is rejected, but p4 sync -p (which populates a client workspace without updating the server's have list) is accepted.
|
p4d -D readonly
p4d -D none
lbr.replication
|
Replica servers can be configured to automatically reject user commands that attempt to modify archived depot files (the "library").
In -D readonly mode, the Perforce Server accepts commands that read depot files, but denies commands that write to them. In this mode, p4 describe can display the diffs associated with a changelist, but p4 submit is rejected.
In -D none mode, the Perforce Server denies any command that accesses the versioned files that make up the depot. In this mode, a command such as p4 describe changenum is rejected because the diffs displayed with a changelist require access to the versioned files, but p4 describe -s changenum (which describes a changelist without referring to the depot files in order to generate a set of diffs) is accepted.
|
|
|
As with the Perforce Proxy, you can use P4TARGET to specify the master server to which a replica server points when retrieving its data.
You can set P4TARGET explicitly, or you can use p4 configure to set a P4TARGET for each named replica server.
A replica server with P4TARGET set must have both the -M and -D flags, or their equivalent db.replication and lbr.replication configurables, correctly specified.
|
|
|
Use the startup.n (where n is an integer) configurable to automatically spawn multiple p4 pull processes on startup.
|
|
|
|
Perforce's p4 pull command provides the most general solution for replication. You can use
p4 pull to configure a replica server that:
|
•
|
uses the startup.n configurable to automatically spawn as many p4 pull processes as required. A common configuration for a warm standby server is one in which one p4 pull process is spawned to replicate the master server's metadata, and multiple p4 pull processes are spawned to run in parallel, and continually update the replica's copy of the master server's versioned files.
|
Although you can run p4 pull from the command line for testing and debugging purposes, it's most useful when controlled by the
startup.n configurables, and in conjunction with named servers, service users, and centrally-managed configurations.
Metadata replication with p4 replicate works by using the same data as Perforce's backup and restore features. The
p4 replicate command maintains transactional integrity while reading transactions out of an originating server and copying them into a replica server's database.
Unlike p4 pull, the
p4 replicate command is capable of replicating
only server metadata. While the
p4 pull command, when applied to metadata, replicates all server metadata,
p4 replicate, is capable of filtering master server's metadata on a table-by-table basis with the
-T flag. If you need to use
p4 replicate's filtering capabilities but also require access to the versioned files, you must provide your own independent mechanism (for example,
p4 pull -u, or
rsync) by which the replica can access the versioned files.
Replica servers are supplied with metadata from a master server by means of the p4 replicate command. The
p4 replicate command polls the master server for new journal entries and outputs them on standard output or pipes the journal entries to a subprocess (such as
p4d -jrc) specified on the
p4 replicate command line. When
p4 replicate is running, start a
p4d for the replica Perforce Server that points to the database into which the journal records are being restored. Users can connect to the replica server the same way they connect to the master server.
p4d -u serviceuser -M readonly -D none -r
replicaroot -p
replicaport
In most cases, the command supplied to
p4 replicate is a variation of
p4d -jr, which reads the journal records from the master server into a set of database files used by the replica server.
A typical invocation of p4 replicate looks like this:
p4 replicate -s statefile -i
interval [-k -x -R] [-J
prefix] [-T
tables]
command
The following table describes commonly-used flags for the replicate command. For a complete list of flags, see the
Perforce Command Reference.
|
|
|
|
|
Specify the polling interval, in seconds. The default is two seconds. To disable polling (that is, to check once for updated journal entries and then exit), specify an interval of 0.
|
|
|
|
|
|
Keep the pipe to the specified command subprocess open between polling intervals.
By default, p4 replicate shuts the pipe down between polling intervals. If you are using -k, you must use the -jrc option to check consistency. If you are not using -jrc, do not use -k to keep the connection open. (For details, see p4 replicate journal-processing commands)
|
|
|
Specify the name of the state file. The state file is a one-line text file that determines where subsequent invocations of p4 replicate start reading data. The format is journalno/byteoffset, where journalno is the number of the most recent journal, and byteoffset is the number of bytes to skip before reading. If no byte offset is specified in the file, replication starts at the beginning of the specified journal file. If no state file exists, replication starts with the first available journal, and a state file is created.
|
|
|
Configures p4 replicate to exit when journal rotation is detected This option is typically used in offline checkpointing configurations. By default, p4 replicate continues to poll the master server until it is stopped by the user.
|
|
|
A command to process the journal records; see p4 replicate journal-processing commands.
|
A typical command supplied to
p4 replicate looks like this:
p4d -r replicaroot -f -b 1 -jrc -
This invocation of p4d makes use of three replication-specific flags to
p4d:
The -jrc flag instructs
p4d to check for consistency when reading journal records. Batches of journal records are read in increasing size until
p4d processes a marker which indicates that all transactions are complete. After the marker is processed, the affected database tables are locked, the changes are applied, and the tables are unlocked. Because the server always remains in a state of transactional integrity, it is possible for other users to use the replica server while the journal transactions are being applied from the master server.
The -b 1 flag refers to bunching journal records, sorting them, and removing duplicates before updating the database. The default is 5000 records per update, but in the case of replication, serial processing of journal records (a bunch size of 1) is required; hence, each line is read individually.
The -f flag supplied to
p4d in conjunction with the
-jr flag forces
p4d to ignore failures to delete records. This flag is required for certain replication configurations because some tables on the replica server (depending on use cases) will differ from those on the master server.
To set a Perforce server name, set the P4NAME environment variable or specify the
-In command line flag to
p4d when you start the server. Assigning names to servers is essential for configuring replication. Assigning server names permits most of the server configuration data to be stored in Perforce itself, as an alternative to using startup flags or environment values to specify configuration details. In replicated environments, named servers are a necessity, because
p4 configure settings are replicated from the master server along with other Perforce metadata.
You can use p4 configure on the master to control settings on
both the master and the replica, because configuration settings are part of a Perforce server's metadata and are replicated accordingly.
After the configuration data has been replicated, the two servers have different server monitoring levels. That is, if you run
p4 monitor show against
master:11111, you see both active and idle processes, because for the server named
master, the
monitor configurable is set to
2. If you run
p4 monitor show against
replica:22222, only active processes are shown, because for the
Replica1 server,
monitor is
1.
A standard user is a traditional user record used for human or automated system access. A
service user is used for server-to-server authentication, as part of the replication process. Creating a service user for each master, replica, or proxy server greatly simplifies the task of interpreting your server logs. Service users can also help you improve security, by requiring your replica servers to have valid login tickets before they can communicate with the master server. Service users do not consume Perforce licenses.
User: service1
Email: [email protected]
FullName: Service User for Replica Server #1
Type: service
|
By default, the output of p4 users omits service users. To include service users, run
p4 users -a.
A newly-created service user that is not a member of any groups is subject to the default ticket timeout of 12 hours. To avoid issues that arise when a service user's ticket ceases to be valid, create a group for your service users that features an extremely long timeout. On the master server, issue the following command:
Add service1 to the list of
Users: in the group, and set the
Timeout: to a large value (in this example, approximately 63 years)
Group: service_users
Timeout: 2000000000
PasswordTimeout: unset
Subgroups:
Owners:
Users:
service1
|
On the master server, use p4 protect to grant the service user
super permission. Service users are tightly restricted in the commands they can run, so granting them
super permission is safe.
When you start a replica that points to a master server with P4TARGET, you must specify both the
-M (metadata access) and a
-D (depot access) flags, or set the configurables
db.replication (access to metadata) and
lbr.replication (access the depot's library of versioned files) to control which Perforce client commands are permitted or rejected by the replica server.
Set P4TARGET to the the fully-qualified domain name or IP address of the master server from which a replica server is to retrieve its data. You can set
P4TARGET explicitly, specify it on the
p4d command line with the
-t host:port flag, or you can use
p4 configure to set a
P4TARGET for each named replica server. .
If you specify a target, p4d examines its configuration for
startup.n commands: if no valid
p4 pull commands are found,
p4d runs and waits for the user to manually start a
p4 pull command. If you omit a target,
p4d assumes the existence of an external metadata replication source such as
p4 replicate.
Where n represents the order in which the commands are executed: the command specified for
startup.1 runs first, then the command for
startup.2, and so on. The only valid startup command is
p4 pull.
If you are using p4 pull for both metadata and
p4 pull -u for versioned files, start your replica server with
p4d -t host:port -Mreadonly -Dreadonly. Commands that require read-only access to server metadata and to depot files will succeed. Commands that attempt to write to server metadata and/or depot files will fail gracefully.
|
•
|
p4 replicate only replicates metadata. If you are using p4 replicate and also require the replication of (or access to) archive files, the archive files need to be made available to the replica Perforce Server by independent means.
|
If you are using p4 pull for metadata-only replication (that is, if you have no
p4 pull -u commands configured to replicate depot contents), start the server with
p4d -t host:port -Mreadonly -Dnone. You must specify a target.
This example illustrates the use of p4 replicate to replicate metadata between a master server and a replica server. The replica server will hold a replica of the master server's metadata, but none of the versioned files. This configuration is useful for offloading server-intensive report generation, and for performing offline checkpoints; more generalized scenarios require replicas based on the
p4 pull command.
|
5.
|
Install a license file in the P4ROOT directory on the replica server. Contact Perforce Technical Support to obtain a duplicate of your master server license file.
|
|
6.
|
p4 replicate keeps track of its state (the most recent checkpoint sequence number read, and a byte offset for subsequent runs) in a statefile.
|
The -M readonly flag ensures that commands that read metadata are accepted, but blocks commands that write to metadata. Because this server has no versioned files, commands that access depot files are blocked by
-D none.
The replica2 server can be stopped, and checkpoints can be performed against it, just as they would be with the master server. The advantage is that the checkpointing process can now take place without any downtime on the master server.
Users connect to replica servers by setting P4PORT as they would with any other server.
Commands that require access to versioned file data (p4 sync, for example) fail, because this configuration replicates only the metadata but not the versioned files in the depot.
To learn more about p4 replicate, see "Perforce Metadata Replication" in the Perforce Knowledge Base:
|
•
|
Your master server is named Master and is running on a host called master, using port 11111, and its server root directory is /p4/master
|
|
•
|
Your replica server will be named Replica1 and will be configured to run on a host machine named replica, using port 22222, and its root directory will be /p4/replica.
|
|
|
You cannot define P4NAME using the p4 configure command, because a server must know its own name to use values set by p4 configure.
You cannot define P4ROOT using the p4 configure command, to avoid the risk of specifying an incorrect server root.
|
To define the behavior of the replica, you enter configuration information into the master server's
db.config file using the
p4 configure set command. Configure the master server first; its settings will be replicated to the replica later.
|
1.
|
To set the server named Replica1 to use master:11111 as the master server to pull metadata and versioned files, issue the command:
|
|
3.
|
Set the Replica1 server configurable to 1, which is equivalent to specifying the " -vserver=1" server startup flag:
|
Each p4 pull -u command creates a separate thread for replicating archive data. Heavily-loaded servers might require more threads, if archive data transfer begins to lag behind the replication of metadata. To determine if you need more
p4 pull -u processes, read the contents of the
rdb.lbr table, which records the archive data transferred from the master Perforce server to the replica. To display the contents of this table when a replica is running, run:
If rdb.lbr indicates a large number of pending transfers (that is, many rows of records), consider adding more "
p4 pull -u"
startup.n commands to address the problem.
|
6.
|
Set the db.replication (metadata access) and lbr.replication (depot file access) configurables to readonly:
|
|
9.
|
Set the Replica1 configurable for the serviceUser to service.
|
|
•
|
A target server of master:11111, which is equivalent to specifying -t master:11111 on the command line
|
|
•
|
Both db.replication and lbr.replication set to readonly, which is equivalent to specifying -M readonly -D readonly on the command line
|
|
•
|
A series of p4 pull commands configured to run when the master server starts
|
To name your server Replica1, set
P4NAME or specify the
-In flag and start the replica as follows:
When the replica starts, all of the master server's configuration information is read from the replica's copy of the
db.config table (which you copied earlier). The replica then spawns three
p4 pull threads: one to poll the master server for metadata, and two to poll the master server for versioned files.
To confirm that the p4 pull commands (specified in
Replica1's
startup.n configurations) are running , issue the following command:
If the p4 pull and/or
p4 pull -u processes are terminated, read-only commands will continue to work for replica users as long as the replica server's
p4d is running,
|
|
Replica servers in -M readonly -D readonly mode will retrieve versioned files from master servers even if started without a p4 pull -u command to replicate versioned files to the replica. Such servers act as "on-demand" replicas.
Administrators: be aware that creating an on-demand replica of this sort can affect server performance or resource consumption, for example, if a user enters a command such as " p4 print //...", which reads every file in the depot.
|
When you copied the versioned files from the master server to the replica server, you relied on the operating system to transfer the files. To determine whether data was corrupted in the process, run
p4 verify on the replica server:
Any errors that are present on the replica but not on the master indicate corruption of the data in transit or while being written to disk during the original copy operation. (Run
p4 verify on a regular basis, because a failover server's storage is just as vulnerable to corruption as a production server.)
You can perform all normal operations against your master server (p4 -p master:11111 command). To reduce the load on the master server, direct reporting (read-only) commands to the replica (
p4 -p replica:22222 command). Because the replica is running in
-M readonly -D readonly mode, commands that read both metadata and depot file contents are available, and reporting commands (such as
p4 annotate,
p4 changes,
p4 filelog,
p4 diff2,
p4 jobs, and others) work normally. However, commands that update the server's metadata or depot files are blocked.
Some scenarios are relatively straightforward: consider a command such as p4 sync. A plain
p4 sync fails, because whenever you sync your workspace, the Perforce Server must update its metadata (the "have" list, which is stored in the
db.have table). Instead, use
p4 sync -p to populate a workspace without updating the have list:
Some commands affect metadata in more subtle ways. For example, many Perforce commands update the last-update time that is associated with a specification (for example, a user or client specification). Attempting to use such commands on replica servers produces errors unless you use the
-o flag. For example,
p4 client (which updates the
Update: and
Access: fields of the client specification) fails:
However, p4 client -o works:
If you are using a replica server as the foundation of a build farm, the client workspace specifications must be created on the master Perforce server first, so that they can be replicated and available to users of the replica.
If a command is blocked due to an implicit attempt to write to the server's metadata, consider workarounds such as those described above. (Some commands, like
p4 submit, always fail, because they attempt to write to the replica server's depot files; these commands are blocked by the
-D readonly flag.)
You can use the P4Broker with a replica server to redirect read-only commands to replica servers. This approach enables all your users to connect to the same host:port setting (the broker). In this configuration, the broker is configured to transparently redirect key commands to whichever Perforce Server is appropriate to the task at hand.
In a replicated environment, the master server and replica server(s) must be at the same release level: whenever you upgrade a master server, you must upgrade all replica servers.
Because p4 replicate relies on journal records to transfer metadata, and because it supports filtering of database tables, the process of upgrading a replica server in a
p4 replicate environment can be somewhat complex.
To upgrade a server in a p4 replicate environment:
At this point, the replica server (having completed its upgrade) will receive any new transactions that completed on the master server between the replica server's shutdown and the replica server's restart. Because journaling was disabled during the master server's upgrade, the replica server does not receive redundant journal records pertinent only to the master server's upgrade. Because the replica server was upgraded first, it is able to process records from both the pre-upgrade and post-upgrade master server.
|
•
|
Large numbers of "Perforce password (P4PASSWD) invalid or unset" errors in the replica log indicate that the service user has not been logged in or that the P4TICKETS file is not writable.
|
In the case of a read-only directory or P4TICKETS file,
p4 login appears to succeed, but
p4 login -s generates the "invalid or unset" error. Ensure that the
P4TICKETS file exists and is writable by the replica server.
In a p4 replicate environment, you can prevent submission of changes by setting permissions on the replica's versioned files to read-only (relative to the userid that owns the replica
p4d process.) To enforce this requirement, you can use trigger scripts on both the master and the replica server, written to succeed on the master server and fail on the replica server.
In an environment based solely on p4 pull and
p4 pull -u, this restriction is automatically enforced: the settings of the
-D and
-M flags (or the corresponding
lbr.replication and
db.replication configurables) ensure that replica servers reject commands that attempt to modify the metadata or depot files.
Copyright 1997-2011 Perforce Software.
