June 6, 2014

4 Replica Monitoring Tools Every Server Admin Should Know

Traceability

Monitoring and maintaining replica integrity

For administrators of complex installations with multiple deployed replicas, keeping track of the status of the master and each of its replicas can be a significant challenge, and this challenge has only grown with the introduction of Edge Server functionality.

To help you cope with the increasing complexity of the task, we have been steadily improving the power of the tools at your disposal.

We've also introduced a new manual: Distributing Perforce:

Distributing Perforce is a guide intended for administrators responsible for installing, configuring, and maintaining multiple interconnected or replicated Perforce services. Administrators of sites that require only one instance of the Perforce service will likely find the Perforce System Administrator's Guide sufficient.

In this article, I'd like to draw your attention to several changes that we made in the 2013.2 release, which you might have overlooked:

  1. The integrity.csv server log
  2. Modifications and enhancements to the journaldbchecksums command
  3. Changelist checksumming
  4. New configurables

So sit down, pour yourself a cup of tea, and dig in, because there's a fair amount of things to cover here.

Our goal, with respect to server integrity, is to give you a set of tools which can be configured and used as a cohesive set:

  • always-on: continually, routinely, and automatically checked
  • historically tracked for long-term analysis and post-mortem study
  • safe to use in a production environment (low impact on performance and concurrency)
  • reliably useful to harried server administrators: few false positives, actionable information about server problems.

The changes that we made in the 2013.2 release are in addition to the collection of replica monitoring tools already present:

  • p4 pull -l -j
  • p4 pull -l -s
  • p4 verify
  • p4 dbverify
  • p4d -xx

The 2013.2 release also includes a complete overhaul of the p4 journaldbchecksums command. That command, introduced in the 2011.1 release, has been useful for administrators at large sites, but has suffered from various limitations.

As you'll see, the new implementation of the p4 journaldbchecksums command is much more sophisticated.

Now, let's get to those details!

Integrity.csv

If you are running a replica server at release 2013.2 or higher, please consider configuring an integrity log. This can be as simple as:

p4 configure set MyReplicaId#serverlog.file.1=integrity.csv

although the configuration can be more complex if necessary for your environment.

Once it is defined, the integrity log will contain log records of a new record type. The new log record includes information about major events that occur during replica integrity checking, such as:

The checksum for db.rev was checked, and did not match.

Having an integrity.csv log defined ensures that your replica servers are capturing important integrity issues in a long-term log file which is in a format that can be parsed, searched, and processed by administration tools.

This means that it becomes much easier to answer questions like:

When did the checksum for db.rev first start to show problems on replica Berlin-4?

Certain configurables (described farther below) increase the amount of data that is logged to integrity.csv.

The nice thing about the integrity.csv log is that it only contains information about replica integrity events. You don't have to go searching through your main server logs for these messages anymore. Moreover, the log records in integrity.csv are detailed, including time stamps and other system information, and the log file is automatically rotated at each journal rotation, so that it's easy to match up incidents in your integrity.csv logs with the other logs and journals kept by your servers.

Except in highly unusual situations, the integrity.csv log file is a low-volume, compact log, and it should be straightforward to keep these logs for a long period of time to preserve the historical record.

Enhancements to the journaldbchecksums command

In releases 2011.1 through 2013.1, the journaldbchecksums command was fairly simple. As of this release, the command becomes substantially more complex; there are now 4 basic variants of the journaldbchecksums command:

  • p4 journaldbchecksums [-t tableincludelist | -T tableexcludelist] [-l N]
  • p4 journaldbchecksums -u filename -t tablename [-v N] [-z]
  • p4 journaldbchecksums -s -t tablename [ -b blocksize ][-v N]
  • p4 journaldbchecksums -c change

Specifying table sets

The first form of the command is the traditional journaldbchecksums command from release 2011.1. This form of the command is used for monitoring whether the BTree-level table checksum for a particular table matches across various servers.

This form of the command remains one of the first levels of defence for a server administrator; if the table checksums are matching, you have ironclad evidence that your replica holds the correct data.

But table level checksum matching works better for some tables than for others, so the journaldbchecksums command now has various flags that make it easier to control which tables have checksum monitoring. You can now specify particular lists of tables, but since that's rather onerous, we've classified all the database tables into three groups, which you can refer to using the '-l' ("level") flag:

  • Level 1 corresponds to the most important system and revision tables,
  • Level 2 includes all of level 1 as well as certain metadata that is not expected to differ between replicas, and
  • Level 3 includes all metadata, including metadata that is likely to differ between replicas, particularly build farms and edge servers.

For a pure read-only replica, used for example for disaster recovery, checking checksums at level 3 is valuable, for it gives you the highest level of confidence that you could use that replica if a disaster should occur.

For an Edge Server, checking checksums at level 2 is valuable, to ensure that the Edge Server is delivering the correct results for all metadata that should be globally consistent.

For a Build Server in your build farm, checking checksums at level 1 is likely sufficient.

Unloading a single table

The second form of the command provides a new tool for use in situations where you suspect a data problem, and where the table in question is relatively small.

When the -u flag is used, the journaldbchecksums command co-ordinates the dumping of a single table by the master server and by all connected replicas, so that the table is dumped at the same "logical time" and hence its contents can be usefully diff'd.

This works by having the master server dump the table and write a journal note describing the dump; when the replica encounters this note, it too will dump the table and then the table dumps can be diff'd.

You can request that the table dump be in compressed format, which may make it easier to transport it to another site for diff'ing, and you can specify a server version number to downgrade your diff to an older version, similarly to running 'p4d -jd -p N'.

Unloading a table is not a routine practice. Rather, this is a tool that you might use, with the assistance of Perforce Technical Support, to dig into the details of a table that seems to be unexpectedly inconsistent.

Incrementally scanning a table

The third form of the command provides a sub-table, incremental diff feature, designed for use with tables that are too large to unload and diff as is.

When the -s flag is provided, the server will scan the table, a block at a time, locking and unlocking the table between blocks. For each block of data, the server will compute a 'block checksum' and write that as a journal note. When the replica encounters that journal note, it will read the corresponding block of data from its table and confirm that the blocks match.

A 'p4 journaldbchecksums -s' command invocation is intended to be used in coordination with Perforce Technical Support, as one of a number of tools that can be deployed to assess the severity of damage that has occurred in an incident.

You will probably never use 'p4 journaldbchecksums -s'. But it's good to know it's there, just in case.

Changelist checksumming

The '-c' flag to the journaldbchecksums command is used with changelist checksumming, described below.

Changelist checksums

For many years, the 'submit' command has computed a checksum of the changelist being submitted.

This is a fairly simple checksum; it includes a list of the file names and their revision numbers from the changelist.

The original purpose of the changelist checksum was to detect various race conditions, such as an accidental attempt to revert some or all of the files out of a changelist by running 'p4 revert' simultaneously with 'p4 submit'. It still serves that purpose, but since it is a first-order summary of the "contents" of a single changelist, we are now also using it for replica integrity verification.

As of 2013.2, the master server can be configured so that, when each changelist is submitted, the server also writes a journal note with the changelist checksum.

When the replica encounters this journal note, it then re-computes the checksum against its database and verifies that they match.

Additionally, the '-c flag' of the journaldbchecksums command allows you to explicitly re-compute the checksum of a change, and write a new journal note with that checksum, for after-the-fact verification of a older change.

Since the changelist checksum only examines a subset of the overall metadata of a changelist, there are certain sorts of damage that it may miss.

Still, since it is already computed by the submit command, it is a very low-impact, incremental, always-on bit of integrity checking that can add additional coverage to your overall process.

Making your replica checksumming "always on"

Prior to release 2013.2, you had to remember to issue the p4 journaldbchecksums command periodically, or you had to write an automated tool to issue it for you periodically, and you had to remember to check the replica log right away so that you could find the results of the command.

In release 2013.2, we've made that process much easier.

The 2013.2 release adds several new configurables:

rpl.checksum.change

The rpl.checksum.change configurable can be set to 0,1,2,3; it defaults to 0.

  • 0 means: don't do any changelist checksumming
  • 1 means: submit command writes journal note at end of submit
  • 2 means: replica should verify the changelist summary, and should write to integrity.csv if the changelist does not match
  • 3 means: replica should write the results to integrity.csv even if the changelist DOES match

Thus just defining an integrity.csv log and setting the rpl.checksum.change configurable to 2 gets you always-on low-impact changelist checksumming for all your replicas.

rpl.checksum.table

The rpl.checksum.table configurable can be set to 0,1,2; it defaults to 0.

  • 0 means: table-level checksumming only
  • 1 means: journal notes for table-unload and table-scan are processed by the replica, and are logged if the check fails
  • 2 means: results of journal note processing in the replica are logged even if the results match.

I think that for most sites that have one or more replicas, setting rpl.checksum.table=1 would be a good setting.

rpl.checksum.auto

The rpl.checksum.auto configurable can be set to 0-3; it defaults to 0.

Setting rpl.checksum.auto causes your server to automatically perform a journaldbchecksums -l N at the completion of every journal rotation. The command will include more or fewer tables in the checksumming, depending on the value of the configurable, which you can read about in the Distributing Perforce manual.

I think that, for most sites, rpl.checksum.auto=1 would be a good setting.

Summing it all up

If you run one or more replica servers at your installation, and if you are at release level 2013.2 or higher, here is my suggestion:

  • Configure the integrity.csv log file for each of your replicas
  • Configure rpl.checksum.change=2 to automatically verify the checksums on each submitted changelist as it is replicated to each replica
  • Configure rpl.checksum.table=1 to ensure that table checksum mismatches are written to the integrity.csv log
  • Configure rpl.checksum.auto=1 to ensure that your most critical database tables are automatically checksummed at every journal rotation point
  • Keep an eye on your integrity.csv files, and if they show any problems, don't hesitate to contact Perforce Technical Support for assistance in figuring out what happened and how to repair it.

Thanks for listening, and do drop us a line to let us know how things are going!