Backups and restoration

Backups are available as part of the Helix TeamHub packages. Backups are activated with configuration flags. The nature of backups changes from one TeamHub component to another:

Component	Method	Schedule	Description
Assets	Archival	Daily at 00:00	Attachments, Avatars, Logs, Configuration files
MongoDB	Archival	Daily at 00:00	Mongo dumps
Repositories	Incremental sync	Daily at 02:00	Repository type specific tools: git, svnsync, hg
Docker Registry	Archival	Daily at 04:00	Docker images

Preparation

TeamHub stores backups at /var/opt/hth/backups. For production deployments, the recommendation is to mount a reliable external storage that has enough space to accommodate at least 1.5 times the TeamHub data set (/var/opt/hth/shared).

Enabling backups

To enable backups, activate them by setting the configuration flags as shown below:

Combo

1. Merge the following configuration to /var/opt/hth/shared/hth.json and make sure you add the backup settings under the existing keys if they already exist in the configuration.

   {
     "backend": {
       "backups": true
     },
     "mongodb": {
       "backups": true
     },
     "repos": {
       "backups": true
     },
     "docker_registry": {
       "backups": true
     }
   }
   

2. Apply the changes by reconfiguring TeamHub:

sudo hth-ctl reconfigure

Cluster and High Availability

In Cluster and High Availability deployment, the TeamHub services are distributed across the server roles. To enable backups, activate the backup flags in /var/opt/hth/shared/hth.json on the appropriate servers. Make sure you add the backup settings under the existing keys if they already exist in the configuration.

Helix TeamHub DB

{
  "mongodb": {
    "backups": true
  }
}

By default, mailing is not configured for the DB node. To get notifications if failures occur during DB backups, you need to add the following:

1. Configure the postfix section of hth.json to be identical with the one on your web node.

2. Add the email that will receive the notifications to the app section of hth.json:

   {
     "app": {
       "email": "[email protected]"
   
     }
   }
   

Helix TeamHub Web

{
  "backend": {
    "backups": true
  },
  "repos": {
    "backups": true
  },
  "docker_registry": {
	"backups": true
  }
}

Apply the changes by reconfiguring TeamHub on each server:

sudo hth-ctl reconfigure

Configuring how many backups to keep before oldest backup gets removed

You can configure archival mode backups to prune old backups. The configuration below keeps the 30 latest backup archives and deletes the oldest when a new backup occurs.

{
  "backups": {
    "keep": 30
  }
}

A separate setting exists for the Docker Registry:

{
  "docker_registry": {
    "backups_keep": 5
  }
}

Unlike with other backups, the Docker Registry backups_keep value should be low because images tend to take up a lot of space, even if they are compressed.

Restoring backups

Because TeamHub backups are modular, when it comes to restoring the system from a backup, it is important to consider the following:

• Follow the correct order of restoring the backup components:

  1. Assets
  2. Database
  3. Repositories
• Because backups for TeamHub components are taken daily, it is important to restore all components from the same day. Doing otherwise will lead to data inconsistency.

Stopping services

Before starting the restoration process, it is a good idea to stop all the TeamHub services:

sudo hth-ctl stop

Restoring assets

1. SSH into one of the TeamHub Web servers (or the Combo) and switch to the hth user.
2. Copy the backend_backup.tar from backup storage, which is located at /var/opt/hth/backups/backend_backup/<date>/.

3. Extract the archives and restore:

   sudo su - hth
   tar xvf backend_backup.tar
   cd backend_backup/archives; ls *.tar.gz | xargs -i tar xvf {} -C /
   

Restoring MongoDB databases

1. SSH into the TeamHub DB server (or the Combo) and switch to the hth user.
2. Copy mongodb_backup.tar from the backup storage /var/opt/hth/backups/mongodb_backup/<date>/.

3. Start MongoDB, extract the archives, and restore:

   sudo su - hth
   sudo hth-ctl start mongodb
   tar xvf mongodb_backup.tar; cd mongodb_backup/databases/; tar xvf MongoDB.tar.bz2
   

   For Combo deployment without MongoDB authentication, use the following command to restore the database:

   mongorestore --port 4002 --drop MongoDB/

   For Cluster, HA or Combo deployments with MongoDB authentication, use admin credentials to restore the database:

   mongorestore --port 4002 -u <admin-username> -p <admin-password> --drop MongoDB/

Restoring repositories

Repositories are stored under the /var/opt/hth/shared/companies/<company>/projects/<repo_type>/repositories directory. To restore backups from the backups store:

1. SSH into one of the TeamHub Web servers (or the Combo) and switch to the hth user.

2. Run the following script:

   #!/bin/bash
   
   companies=`ls /var/opt/hth/backups/repos_backup/var/opt/hth/shared/companies/`
   # Loop through companies
   for company in $companies; do
     # Loop through projects
     projects=`ls /var/opt/hth/backups/repos_backup/var/opt/hth/shared/companies/$company/projects/`
     for project in $projects; do
       echo "Restoring repositories for project $project in company $company"
       repos_dest="/var/opt/hth/shared/companies/$company/projects/$project/repositories/"
       if [ ! -d $repos_dest ]; then
         # Company/Project may have been renamed
         echo "Creating $repos_dest"
         mkdir -p $repos_dest
       fi
       rsync -av --delete /var/opt/hth/backups/repos_backup/var/opt/hth/shared/companies/$company/projects/$project/repositories/ $repos_dest
     done
   done
   

3. To restore the repositories hooks, execute the following commands to regenerate them:

   sudo su - hth
   sudo hth-ctl start redis
   cd /opt/hth/application/backend/current
   rake hth:restore:all
   

Restoring Docker Registry

1. SSH into one of the TeamHub Web servers (or the Combo) and switch to the hth user.
2. Copy docker_registry_backup.tar from the backup storage /var/opt/hth/backups/docker_registry_backup/<date>/.

3. Extract the archives and restore:

   sudo su - hth
   rm -rf /var/opt/hth/shared/storage/docker_registry/docker/
   cd /var/opt/hth/backups/docker_registry_backup/<date>
   tar xvf docker_registry_backup.tar
   cd docker_registry_backup/archives; ls *.tar.gz | xargs -i tar xvf {} -C /
   

Starting TeamHub

Start back all TeamHub services:

sudo hth-ctl start

Reconfiguring TeamHub

If there have been changes to the hth.json restored from backups, apply those changes by running the reconfigure command:

sudo hth-ctl reconfigure