Reclaim disk space by archiving files

Over time, Helix Core Server can accumulate many revisions of files. Some files might be from projects that are no longer active. Therefore, you might want to reclaim the disk space. One way is by transferring those versioned files into an archive depotClosed A special depot into which versioned files (also known as archive files) can be transferred.. Archive depots free up disk space by moving files that are rarely accessed to bulk storage.

Compare the advantages of using an archive depot and p4 archive to the p4 delete and p4 obliterate commands:

p4 delete

p4 archive

p4 obliterate

  • no disk space reclaimed

  • file history is preserved and "deleted" files can be restored

  • merely marks files as deleted at the head revision

  • disk space reclaimed on the storage used for your local depot

  • files moved to the archive storage

  • file history is preserved and archived files can be restored

  • disk space reclaimed

  • obliterated files cannot be restored

  • permanently removes file and its history.

Task streams must be unloaded with the p4 unload command prior to the task stream file revisions being archived to an archive depot.

Transferring versioned files to an archive depot is not allowed if the files are in a remote depot or are already in an archive depot.

In a commit-edge environment, the archive depot is not replicated between the commit and Edge Servers.

Transfer files into an archive depot

  1. Create a new depot by using the p4 depot command.
    p4 depot -t archive ArchiveDepot

    where ArchiveDepot is the name for your new depot of type archive.

  2. Set the archive depot’s Map field to an absolute path where the archived files will be stored. The path can be locally mounted network storage, detachable storage, or a mounted block storage device for the Cloud. If you are using S3 storage, see S3 storage for archive depots.
  3. To transfer all unbranched binary files to the archive depot from your existing depot:
    p4 archive -D ArchiveDepot //LocalDepot/...

    where LocalDepot is the name of your standard depot of type local.

    Binary files, compressed and uncompressed, are the default file types that p4 archive transfers. You might want to adjust the behavior of the command, so consider its Options, such as -t to make p4 archive transfer text files. To learn more, see File type modifiers in the Helix Core Command-Line (P4) Reference.

  4. (Optional) Unmount the volume to which the LocalDepot archive files were written, and store it in a secure location.

Restore files from an archive depot

  1. Mount the volume on which the archive depot’s files are stored.
  2. Make sure that the path matches the Map field of the archive depot.
  3. Use the p4 verify -A command to verify files before you restore them.

    For example, p4 verify -A //ArchiveDepotName/... checks for corrupt or missing files.
    Files archived into this depot have their original path but with the depot name.
    For example, //ArchiveDepotName/LocalDepot/Mainline/... includes the LocalDepot from which the files were transferred to the archive depot.

  4. Use p4 restore to transfer the files from the archive depot to the original local depot.
    p4 restore -D ArchiveDepotName //LocalDepot/...
  5. (Optional) unmount the volume from which the archive files were restored.

Purge files from an archive depot

Warning

If you purge files from an archive depot, they are permanently deleted.

  1. Mount the volume on which the archive depot’s files are stored.
  2. If you want Helix Core Server to retain the metadataClosed The data stored by Helix Core Server that describes the file revisions in the depot, where they get their content from (see lazy copy), and the current state of client workspaces, protections, groups, users, labels, streams, and branches. Metadata is stored in the server database and is separate from the archive files that users submit. of the files you are going to purge, see the -p option of the p4 archive command.

  3. Use p4 archive -p to purge files.

  4. (Optional) unmount the volume from which the archive files were purged.

S3 storage for archive depots

Your organization might find it economical to store archive depots by using:

Note

Although this information focuses on archive depots, this feature is supported for depots of all types except remote depots, provided the depots are stored in the same S3 bucket.

You can create an S3 data source as a place to store archive files for a Helix Core Server depot. Create an archive depot specification that sets the Address field to map to the S3 datasource. The location of the S3 bucket and the credentials are stored in the depot spec on a per-depot basis using the Address field in the following format:

Address: s3,region:us-east-1,bucket:p4d-depot,accessKey:,secretKey:******

where:

  • s3 must be at the start of the address

  • The settings are comma-separated, with a colon to separate the key and value.

The S3 data source requires the following AWS Identity and Access Management (IAM) permissions of the Amazon S3 API Reference: s3:GetObject, s3:PutObject, and s3:DeleteObject.

Supported keys are:

Key

Note

url https://bucketname.s3.amazonaws.com

Some S3 implementations might not require this key.

region Some S3 implementations might not require this key.
bucket Required
accessKey Required
secretKey Required
token Depending on bucket security, this key might be optional.

Create an S3-backed archive depot

  1. Create a new depot by using the p4 depot command, and set the depot’s Type to archive.

    p4 depot -t archive s3ArchiveDepotName
  2. In the depot specification,

    • Do not change the value of the Map field because it will not be used.

    • Create a new field for the address, and enter the s3 information and credentials as a comma-separated list of key:value pairs according to the requirements of your S3 storage provider.

Examples

Different storage providers might have different requirements. For the most complete, correct, and current requirements, check with your storage provider.

AWS:

Address: s3,region:us-west-2,bucket:my-archivebucket,accessKey:******,secretKey:******

Digital Ocean:

Address: s3,url:https://my-archivebucket.sfo3.digitaloceanspaces.com,bucket:my-archivebucket,accessKey:******,secretKey:******

MinIO:

Address: s3,url:https://server:port/my-archivebucket,bucket:my-archivebucket,accessKey:******,secretKey:******

Google Cloud Storage:

Address: s3,url:https://storage.googleapis.com/my-archivebucket,bucket:my-archivebucket,accessKey:******,secretKey:******
Important

Whether you transfer versioned files to an archive depot or not, be aware that Helix Core Server does not back up or restore versioned files. Check with your cloud storage provider about the availability and data resiliency of your versioned files.