Illumio Core 25.1 Administration Guide

Complete the prerequisite steps in Standby PCE Prerequisites.

On the active PCE, generate an API key. This API key is used only while setting up the standby PCE.

Bring the standby PCE to runlevel 2. On any node of the standby PCE, run the following command:

sudo -u ilo-pce illumio-pce-ctl set-runlevel 2

The active PCE can remain at runlevel 5.

On the standby PCE, run the following commands to set up authentication. In username, give the active PCE's API key authentication username. In secret, give the API key secret.

$ export ILO_ACTIVE_PCE_USER_NAME=username
$ export ILO_ACTIVE_PCE_USER_PASSWORD=secret

Link the standby PCE to its active PCE. On the standby PCE, run the following command. Foractive_pce_fqdn:front_end_management_https_port, give the FQDN and port of the current active PCE. The value in --active-pce is not the same as active_pce_fqdn in the configuration file runtime_env.yml.

sudo -u ilo-pce --preserve-env illumio-pce-ctl setup-standby-pce 
--active-pce active_pce_fqdn:front_end_management_https_port

After replication is set up up for the first time, the status of some services, such as the citus_coordinator_service, might be NOT RUNNING for a long time, and the cluster status is stuck in PARTIAL. This is usually because the service is performing a database backup, which can take time depending on network latency, disk IOPS, traffic flow, and traffic data size. To check whether the backup process is running, use the following command:

ps -ef | grep pg

Example output:

pce      84742 73150 18 16:25 ?        00:04:42 
  /var/illumio_pce/external/bin/pg_basebackup -d host=10.31.2.172
  port=5532 -D /var/traff_dir/traffic_datastore -v -P -X stream -c fast
pce      84747 84742  7 16:25 ?       00:01:54 
  /var/illumio_pce/external/bin/pg_basebackup -d host=10.31.2.172 
  port=5532 -D /var/traff_dir/traffic_datastore -v -P -X stream -c fast

Restart services on the active PCE. On any node of the active PCE, run the following command:

sudo -u ilo-pce illumio-pce-ctl cluster-restart

Check to be sure the PCE you are about to promote is actually a standby PCE and that it is at runlevel 2.

sudo -u ilo-pce illumio-pce-ctl active-standby?

The output should say "standby."

Check to be sure the active PCE has failed and is offline. There must not be any data replicating to the standby PCE. On every node of the active PCE, run the following command:

sudo -u ilo-pce illumio-pce-ctl cluster-status

The output should contain STOPPED. Be sure to repeat this command on every node of the PCE.

On the standby PCE, run the following command to promote the standby PCE.

sudo -u ilo-pce illumio-pce-ctl promote-standby-pce

When the active PCE is down, this command promotes this PCE to be the new primary. If the active PCE is not down, the standby PCE will not be promoted, and a message like "Active PCE is still reachable" is generated.

Make sure that DNS recognizes this as the new active PCE FQDN so devices in your network can find the PCE. Make sure that the values for both active_standby_replication and active_pce_fqdn in the configuration file runtime_env.yml are the PCE FQDN of the former standby (new active) PCE. For example, reconfigure the PCE FQDN on load balancers. The steps depend on your devices and configuration. For more information about the PCE FQDN, see Standby PCE Prerequisites.

Check the VEN synchronization status by running the following command:

sudo -u ilo-pce illumio-pce-ctl promote-standby-check

Run the command repeatedly and watch the output to make sure the VEN sync count increases. This indicates that the DNS change is in effect and the new active PCE has been promoted successfully.

The DNS update for the new PCE FQDN can take some time, depending on the DNS TTL value.

When you are ready, connect a new standby PCE to the new active PCE. Repeat the steps in Standby PCE Prerequisites and Set Up a Standby PCE.