PCE Database Backup
This section provides step-by-step instructions for backing up the PCE databases. Before you start, be sure you understand the technical details of the two PCE databases; see About the PCE Databases for information.
Note
The PCE runtime configuration file, runtime_env.yml, is not included in database backups. You must back up this important file separately. See Back Up the PCE Runtime Environment File.
About PCE Database Backup
You use the PCE database command line utility illumio-pce-db-management to back up, migrate, manage failover, and restore the PCE databases.
Important
You must run the PCE database commands as the PCE runtime user ilo-pce
When to Back Up
Follow your organization's backup policies and procedures, including frequency (such as, hourly, daily, or weekly) and retention location (namely, offsite or on a system other than the PCE cluster nodes).
Illumio recommends backing up the PCE databases in the following situations:
Before and after a PCE version upgrade
After pairing a large number of VENs
After updating a large number of workloads (such as, changing workload policy state or applying labels)
After provisioning major policy changes
After making major changes in your environment that affect workload information (such as, IP address changes)
On-demand backups before performing the procedures in this guide
Back Up the Policy Database
Perform these steps to back up all PCE data, such as before upgrading the PCE.
Back Up the Traffic Database
The traffic database dump can be very large, depending on the traffic datastore size. Therefore, the Supercluster database dump on leader and member PCEs does not include the traffic database dump. The following procedure is provided to back up the traffic data separately.
Note
If you have a multi-node traffic database, do not use this procedure for routine backups. In a multi-node traffic database, the procedure in this section is used only for the initial installation of the multi-node database or when adding or removing worker nodes. For routine backups in a multi-node traffic database, use pgbackrest instead. See Using pgbackrest for Traffic Data Backups.
Perform these steps to back up the traffic database only. If you need to back up the traffic flow data, perform this procedure on every region; traffic flow information is unique to every (region) PCE.
On any data node, run the following command:
sudo -u ilo-pce illumio-pce-db-management traffic dump --file <path_to_traffic_backup_file.tar.gz>
In
path_to_traffic_backup_file.tar.gz, include the filename extension.tar.gz.After the command finishes, copy the backup file to a fault-tolerant storage location.
Using pgbackrest for Traffic Data Backups
Instead of using the built-in PCE backup commands, you can use the pgbackrest tool. For example, pgbackrest can be useful if you have dedicated storage for backups, such as NFS network shared storage. If you have a multi-node traffic database, you must use pgbackrest for backups to ensure adequate space and performance.
Hardware Requirements
A shared filesystem such as NFS mount which is mounted on all the PCE nodes is required for pgbackrest to work. Make sure the NFS disk has enough space to store multiple backups. Specify the root location of this mount with the backup_root key in the runtime_env.yaml, shown below in "Enabling pgbackrest."
The NFS mount can be used to store other data in addition to the traffic data. For example, it could store the policy database and runtime_env.yml file. The NFS mount must be a solid-state drive (SSD) disk. Rotational disks cannot be used, because they are too slow for the amount of data involved.
To calculate the size of the NFS mount needed for a multi-node traffic database, use the following formula: Number of worker node pairs x 150 GB x number of days retained + storage needed when occasionally adding or removing a node, which is 400 GB x number of worker node pairs. Optionally, add the amount of storage needed for any additional uses, such as the policy database.
Enabling pgbackrest
To enable the pgbackrest tool, add the following commands to the server runtime_env.yaml, with your cluster values specified where needed:
traffic_datastore_backup_service: pgbackrest_enabled: true backup_destination_type: 'filesystem' backup_root: '<location of NFS root>' backup_encryption_key: '<location of file that contains the backup encryption key>' max_full_backups: '<max number of full backups to retain>' # Defaults to 2
Back Up the Traffic Database (pgbackrest)
Use the following command to take a backup of the traffic database cluster. In a multi-node traffic database, you can run this command on any coordinator or worker node:
sudo -u ilo-pce illumio-pce-db-management traffic cluster-backup
List Available Backups (pgbackrest)
Use the following command to get the list of backups available, in the order in which they were taken:
sudo -u ilo-pce illumio-pce-db-management traffic cluster-backup-list
Restore a Backup (pgbackrest)
Use the following commands to restore data from a given backup. For
backupLabel, substitute the label of the backup to restore:
sudo -u ilo-pce illumio-pce-ctl set-runlevel 1 sudo -u ilo-pce illumio-pce-db-management traffic cluster-restore --backup-label backupLabel
Back Up the PCE Runtime Environment File
The PCE runtime configuration file, runtime_env.yml, is not included in automatic PCE backups. You must manually back up this file to a secure location.
Store a copy of each node's runtime_env.yml file on a system that is not part of the PCE cluster. By default, the PCE Runtime Environment File is located at the following location on each node:
/etc/illumio-pce/runtime_env.yml
If the file is not found there, it has been moved to a custom location. To find the file, check the ILLUMIO_RUNTIME_ENV environment variable.
Important
The runtime_env.yml file contains sensitive information that should be kept secret, such as encryption keys. Take steps to ensure the confidentiality of this file.