p4 journaldbchecksums

Write journal notes with table checksums.

Syntax

p4 [g-opts] journaldbchecksums [-t tableincludelist | -T tableexcludelist] [-l level]
p4 [g-opts] journaldbchecksums -u filename -t tablename [-v version] [-z]
p4 [g-opts] journaldbchecksums -s -t tablename [-b blocksize] [-v version]
p4 [g-opts] journaldbchecksums -c changelist

Syntax conventions

Description

The p4 journaldbchecksums command provides a set of tools for ensuring data integrity across a distributed or replicated installation.

Helix Core Server automatically performs an integrity check when you use the p4 admin checkpoint or p4 admin journal commands, or when you use p4 journaldbchecksums to manually perform an integrity check.

To use this command with structured logging (see p4 logparse), at least one structured log must be capturing events of type integrity.

When an integrity check is performed, the Server writes records to the journal that contains the checksums of the specified tables. If no tables are specified, the Server writes records to the journal that contains the checksums for all tables. Replica servers, upon receiving these records, compare these checksums with those computed against their own database tables, as they would with p4 dbstat. Results of the comparisons are written in the replica’s log.

You can control which tables are checked, by:

  • including individual tables with -t

  • excluding individual tables with -T

  • using one of three levels of verification:

Level Description
1 The most important system and revision tables
2 All of level 1, plus metadata that is not expected to differ between replicas
3 All metadata, including metadata that is likely to differ between replicas, particularly build farms and edge servers

Verification levels are controlled by the rpl.checksum.auto configurable or the -l level option.

When checking individual changelists and individual tables, the rpl.checksum.change and the rpl.checksum.table configurables control when events are written to the log.

See also Verifying replica integrity in the Helix Core Server Administrator Guide.

Options

-b blocksize

When scanning tables, scan blocksize records per block. The default is 5,000. For each block, the server computes a block checksum and writes it as a journal note. Replica servers automatically verify these blocks when processing these notes. This option can be used with large tables on a production system as the table is unlocked between each block. Inspecting the results of the block verifications will reveal the location of damage that affects only part of a database table. See the example of Database Table Block Checksums.

-c changelist

Compute a checksum for an individual submitted changelist. The checksum is written as a journal note, and replica servers automatically verify the checksum of the change when they process these notes. See the example of Changelist Checksums.

-l level

Specify a level for checksumming; each level corresponds to a larger set of tables. These levels correspond to the levels used by the rpl.checksum.auto configurable.

-s -t tablename

Scan the specified database table. See the example of Database Table Unload.

-t tablenames

Specify the table(s) for which to compute checksums. To specify multiple tables, double-quote the list and separate the table names with spaces. The table names must start with "db.". Table names can also be separated by commas. See the example of Database Table Checksums.

-T tableexcludelist

Compute checksums for all tables except those listed.

-u filename -t tablename

Unload to the specified filename the specified tablename. This command also writes a journal note that documents this action and instructs replica servers to unload the same table to the same file when they process this journal note.

Prerequisite: The rpl.checksum.table configurable must be set to 1 or 2 for this option to work.

-v version

When unloading or scanning tables, specify the server version number to use. If no server version number is specified, the current server version is used.

-z

Compress the file when unloading a table.

g-opts

See Global options.

Usage Notes

Can File Arguments Use Revision Specifier? Can File Arguments Use Revision Range? Minimal Access Level Required

N/A

N/A

super

available to an operator user

For more about administering Helix Core Server in distributed or replicated environments, see Helix Core Server Administrator Guide.

Examples

Database Table Checksums

p4 journaldbchecksums [-t tableincludelist | -T tableexcludelist] [-l N]

causes the server to write journal notes containing table checksum information:

p4 journaldbchecksums -t db.rev
@nx@ 12 1487712216 @41@ 9 -933920831 0 4 0 @db.rev@ @@ @@ @@ @@

Edge/Replica servers automatically verify the table checksums when processing these notes, writing the results to the server log and optionally an integrity structured log if configured:

Table db.rev checksums match. 2017/02/21 13:23:36 version 9: expected 0xC8557FC1, actual 0xC8557FC1
p4 logparse -m1 -F f_table=db.rev -T 'f_date f_results' integrity.csv
... f_date 2017/02/21 13:23:36 219149298
... f_results match

The results of table checksum comparison will result in one three possible results:

  • match
  • DIFFER
  • empty
Table db.have checksums DIFFER. 2017/02/21 13:08:38 version 3: expected 0x3BB210EE, actual 0xB1BF3E83

p4 logparse -F f_results=DIFFER -T 'f_date f_table' integrity.csv
... f_date 2017/02/21 13:08:38 203821071
... f_table db.have
Table db.ldap checksums empty. 2017/02/24 11:33:54
version 0: expected 0x0, actual 0x0.

p4 logparse -F f_results=empty -T f_table integrity.csv
... f_table db.ldap

The table checksums might be reported as DIFFER if the database structure diverged as the result of:

Software upgrades:

Some upgrades are performed against a database table when data in a table is accessed.

Replaying a checkpoint or journal:

 

When administrators replay journal data or journal patches using p4d -jr, the transactions replayed into the database are not journaled. This might generate differing checksums.

When replaying journal data in a distributed environment, always use p4d -s -jr so the replayed transactions are journaled, thus enabling downstream edge/replica servers to replay them.

Be aware that p4d -jr run against a replica server only updates the replica's database files and therefore might generate differing checksums.

Journal filtering:

When filtering is active in your replication process, not all journal checksums are expected to match.

To remedy unexpected checksum differences, restore the edge/replica server database from a new checkpoint of the commit/master server.

Changelist Checksums

p4 journaldbchecksums -c change

causes the server to compute a checksum of an individual submitted changelist. This checksum is written as a journal note:

p4 journaldbchecksums -c 12073
@nx@ 15 1487961638 @41@ 12073 1 0 0 0 @46B19358420B468668781A002BA0AC15@ @@ @@ @@ @@

Replica servers automatically verify the checksum of the change when processing these notes and write the results to the integrity structured log:

p4 logparse -F f_change=12073 -T f_results integrity.csv
... f_results match

Server behavior is dependent on the rpl.checksum.change configurable.

Database Table Block Checksums

p4 journaldbchecksums -s -t tablename [ -b blocksize ][-v N]

causes the server to scan the specified database table. The table is scanned in blocks. The number of records in a block is specified by the -b flag, which defaults to 5,000. For each block, the server computes a block checksum and writes it as a journal note:

p4 journaldbchecksums -s -t db.have
@nx@ 17 1487964567 @41@ 3 1 313 0 0 @db.have@ @@@//Talkhouse/build/jar/Talkhouse.jar@@
 @ @@@//Jam/MAIN/src/glob.c@@ @ @2BCDA450287C03DE3433AEB6278EA4AA@ @@

Replica servers automatically verify these blocks when processing these notes and write output to the integrity structured log if configured:

p4 logparse -F 'f_table=db.have' -T 'f_results f_checkSum f_checkSum2' integrity.csv
... f_checkSum 2BCDA450287C03DE3433AEB6278EA4AA
... f_checkSum2 D41D8CD98F00B204E9800998ECF8427E
... f_results failed

This command can be used with large tables on a production system because the table is unlocked between each block. Inspecting the results of the block verifications reveals the location of any damage, which affects only part of a database table.

Database Table Unload

p4 journaldbchecksums -u filename -t tablename [-v N] [-z]

causes the server to unload the specified database table to the specified file. The command also writes a journal note describing this action:

p4 journaldbchecksums -u working.txt -t db.working
@nx@ 16 1487964861 @41@ 10 0 0 0 0 @db.working@ @working.txt@ @@ @@ @@

Replica servers automatically unload the same table to the same file when processing these notes. If only a file name is specified with -u, as in the example above, the unload files are created in the P4ROOT directory of both servers. Any relative path specified with -u is relative to the P4ROOT directory. Absolute paths to the unload file can also be used. Ensure any referenced directory paths exists on both master and replica prior to running the unload.

For a time-consistent comparison of the contents of the table, unload the tables in this way. This command is recommended only for tables that are small. The -z flag specifies that the file should be compressed.