Upgrade the PCE

This section describes how to upgrade the PCE and its UI together. To upgrade the UI alone, see UI-Only Upgrade.

Note

You can upgrade your PCE operating system to RHEL 9 while also upgrading your PCE. See Upgrading to RHEL 9.

When upgrading the PCE and UI packages together, perform these high-level tasks:

Verify that you have met all the Upgrade Prerequisites.
Perform PCE installation planning and prerequisite steps if you have not already performed these tasks on the PCE. See Prepare for PCE Installation.
Back up the PCE.
Download the software.
Stop the PCE.
Install the new PCE and UI.
Update the runtime environment file.
Migrate the database.
Set runlevel to 5.
Verify successful upgrade.

Back Up the PCE

When you are upgrading from a previous PCE version, the first step is to back up your existing data.

Back Up PCE Data

(On an SNC, skip this step.) Before you back up the PCE, determine which data node is running the agent_traffic_redis_server service:

sudo -u ilo-pce illumio-pce-ctl cluster-status

You see the following output:

SERVICES (runlevel: 5) NODES (Reachable: 1 of 1)
====================== ===========================
agent_background_worker_service 192.168.33.90
agent_service NOT RUNNING
agent_slony_service 192.168.33.90
agent_traffic_redis_cache 192.168.33.90
agent_traffic_redis_server 192.168.33.90 <=== run the dump command from this node
agent_traffic_service NOT RUNNING
...

On the data node that is running the agent_traffic_redis_server service, run the following commands:
```
sudo -u ilo-pce illumio-pce-db-management dump --file <location-of-db-dump-file>
sudo -u ilo-pce illumio-pce-db-management traffic dump --file <location-of-traffic-dump-file>
```
In location-of-db-dump-file and location-of-traffic-dump-file enter a file name for the policy database dump and the traffic database dump files, respectively.
Note
On an SNC, run these commands on the single node.
After the dump commands finish, copy the backup files to a fault-tolerant storage location.

Back up the PCE Runtime Environment File

Store a copy of each node's runtime_env.yml file on a system that is not part of the cluster or Supercluster. The default location of the PCE Runtime Environment File is /etc/illumio-pce/runtime_env.yml.

Download the Software

Download the software from the Illumio Support portal (login required).
On the core nodes only, copy the Illumio PCE UI RPM file to the /tmp folder. In the following steps, this file is referred to as illumio_ui_rpm.
On all nodes in the cluster, copy the Illumio PCE software RPM file to the /tmp folder. In the following steps, this file is referred to as illumio_pce_rpm.

Stop the PCE

On all nodes in the cluster, stop the PCE:

sudo -u ilo-pce illumio-pce-ctl stop --wait

On all nodes in the cluster, verify the PCE status is STOPPED:
```
sudo -u ilo-pce illumio-pce-ctl status -sv --wait
```

Install the New PCE and UI

The packages to install depend on the type of PCE node:

Core nodes: Two packages, the PCE RPM and UI RPM.
Data nodes: One package, the PCE RPM.

On each core node in the cluster, log in as root and install the PCE RPM and the UI RPM. Be sure to specify both of the RPM file names on the command line:
```
rpm -Uvh illumio_pce_rpm illumio_ui_rpm
```
For
illumio_pce_rpm
and
illumio_ui_rpm
, substitute the paths and filenames of the two RPM files you downloaded from the Illumio Support portal.
On each data node in the cluster, log in as root and install the PCE RPM:
```
rpm -Uvh illumio_pce_rpm
```
For
illumio_pce_rpm
, substitute the path and filename of the software you downloaded from the Illumio Support portal.
If your PCE also includes an installed NEN, run the following commands:
```
rpm -e illumio_pce_nen --noscripts
rpm -Uvh illumio_pce_rpm --noscripts
rpm -Uvh illumio_nen_rpm
```
For
illumio_pce_rpm
and
illumio_nen_rpm
, substitute the paths and filenames of the two RPM files you downloaded from the Illumio Support portal.
Then use the following command to check the environment:
```
sudo -u ilo-pce illumio-pce-ctl check-env
```

Update the Runtime Environment File

See "What's New and Changed in This Release" to determine if any changes to the PCE Runtime Environment File (runtime_env.yml) are required to upgrade.

Warning

The cluster_type runtime parameter must be set on all PCE nodes starting in PCE 21.5.0, except on a single node cluster (SNC). If you are upgrading from a version earlier than 21.5.0, be sure to set this parameter. For details about what value to use on each type of node, see Reference: PCE Runtime Parameters.

To make changes to the runtime environment configuration:

On all nodes in the cluster, update the runtime_env.yml file.
On all nodes in the cluster, check the validity of the runtime_env.yml file:
```
sudo -u ilo-pce illumio-pce-ctl check-env
```
If any issues are reported by this command, correct them before moving on to the next step.

Migrate the PCE Database

Ensure that you have upgraded all nodes in the cluster to the same version before you perform these steps. Otherwise, none of the nodes in your cluster will start.

On all nodes in the cluster, start the PCE:

sudo -u ilo-pce illumio-pce-ctl start --runlevel 1

For some upgrades, you might be prompted to upgrade the database PostgreSQL software on one of the data nodes.

At the prompt, enter yes to continue the upgrade.

If you do not see this prompt, go to the next step.

Upgrade needed. The agent database is currently running at postgres 
version 9.6 and needs to be upgraded to version 11.4
Upgrade needed. The traffic database is currently running at postgres 
version 9.6 and needs to be upgraded to version 11.4
The PCE software will now upgrade to a newer version of the postgres 
software from an older version.
You need to migrate the PCE database after this process has finished.
Prior to this upgrade, Illumio recommends you make a backup of the 
relevant data directories:
/var/illumio_pce_data/persistent
The upgrade must not be interrupted, otherwise the data might get 
corrupted and prevent the PCE from starting.
Do you wish to continue with the database upgrade. [yes/no]: yes
Proceeding with database upgrade.
Please wait until the system has reached runlevel 1.

On all nodes in the cluster, verify the PCE status:

sudo -u ilo-pce illumio-pce-ctl status -sv --wait

Verify that the runlevel is 1:

sudo -u ilo-pce illumio-pce-ctl get-runlevel

On any node in the PCE cluster, migrate the database to the latest schema version:
```
sudo -u ilo-pce illumio-pce-db-management migrate
```

Set Runlevel 5

Bring the PCE to runlevel 5, full operation:

sudo -u ilo-pce illumio-pce-ctl set-runlevel 5

Important

If you did not run the illumio-pce-db-management migrate command on the primary database, you cannot bring the node up to runlevel 5 and you cannot start the other nodes in the cluster. If some of the nodes in the cluster are already running, they will shut down until you successfully migrate the database. If you attempt to start the upgraded PCE cluster without migrating the database, this error is displayed:

sudo -u ilo-pce illumio-pce-ctl start
Starting Illumio Runtime STARTING 20.96s
$ 
$ Stopping PCE software: DB migrations mismatch for DB: 
avenger_executor_dev: Missing migrations.

Verify Success

On all nodes in the cluster, verify the PCE status:
```
sudo -u ilo-pce illumio-pce-ctl status -s -v -w
```

On any node, verify the cluster status:

sudo -u ilo-pce illumio-pce-ctl cluster-status -w

When you have a front end load balancer (F5 or DNS), make sure that the load balancer is sending requests to the two core nodes in the cluster.
Log into the PCE web console, pair a VEN, and verify VEN sync status is showing as “Verified” for a few randomly selected workloads. Select a Workload's details page and verify that the Policy Sync section shows “Verified.”
Test that the firewall is correctly configured by connecting to restricted services using telnet. When the PCE is running, connections on TCP port 8300 should be accepted from other PCE hosts but rejected otherwise. Connections on TCP port 8443 should be accepted from all sources (unless modified using the -p option).

Upgrading to RHEL 9

You can upgrade your PCE operating system to RHEL 9 while also upgrading your PCE. When upgrading from a previous operating system and previous PCE release, follow these recommended additional steps.

Upgrade the PCE, using the steps outlined earlier in this topic.
Take a backup of the upgraded PCE. Store the backup in a safe storage.
Upgrade the operating system to RHEL 9.
Install the same PCE version you upgraded to in step 1 on RHEL 9.
Restore the backup that was made in step 2.

For details on steps 2 through 5, see "Upgrade the OS on a Running PCE" in the PCE Administration Guide.

Illumio Core 23.5 Install, Configure, Upgrade

Upgrade the PCE

Note

Back Up the PCE

Back Up PCE Data

Note

Back up the PCE Runtime Environment File

Download the Software

Stop the PCE

Install the New PCE and UI

Update the Runtime Environment File

Warning

Migrate the PCE Database

Set Runlevel 5

Important

Verify Success

Upgrading to RHEL 9

Search results