Skip to main content

Illumio Core 21.5 Install, Configure, Upgrade

Deploy a PCE Supercluster

You can deploy the Illumio Supercluster in several ways:

  • New: You have never deployed a PCE and want to deploy a new Supercluster. See Deploy New Supercluster.

  • Expand: You have already deployed a standalone PCE and want to expand it to a Supercluster. See Expand Standalone PCE to Supercluster.

  • Join: You already have more than one standalone PCE and you want to join them together into a Supercluster. Contact your Illumio Customer Support for assistance.

Deploy New Supercluster

Deploying a new PCE Supercluster follows this general workflow:

  1. Install the leader PCE as a standalone PCE.

  2. Install and configure each member PCE as a standalone PCE.

  3. Initialize the Supercluster leader.

  4. Join members to the Supercluster.

  5. Bring the leader and members to a fully operational state.

  6. Verify that the Supercluster is ready for use.

Note

The sequence of events for deploying a Supercluster is not bound by any time requirements; for example, there is no time limit between initializing a Supercluster leader and joining individual members.

Before You Begin: Runtime Configuration

Before you deploy your PCE Supercluster, be aware of the following runtime_env.yml configurations:

  • The value of the parameter service_discovery_encryption_key in the runtime_env.yml file must be exactly the same on all nodes on all PCEs in your Supercluster.

  • You do not need to configure the public IP addresses of other PCEs under the cluster_public_ips parameter. Supercluster PCEs automatically exchange their configured public IP addresses with each other, which get programmed by the VEN to allow workloads to migrate between PCEs.

Optional

Depending on your deployment environment, you might need to make the following changes to the runtime_env.yml file on each PCE in the Supercluster.

When the nodes of each PCE use multiple IP addresses or they use IP addresses other than the one advertised on the node for communication with other PCEs, such as having a NAT between the PCEs in your Supercluster, configure this optional parameter:

  • supercluster.node_public_ip: The public IP address of this node is advertised to other PCEs in your Supercluster deployment. This IP address must be reachable from all other Supercluster PCEs that you want to join. This parameter must be set on all nodes in each PCE. When your PCE is deployed in a public cloud, such as AWS, this must be a public IP address.

When you configure your GSLB for routing VENs to the appropriate PCE, configure this optional parameter on each node in a PCE:

  • supercluster.fqdn: The PCE responds to this FQDN, instead of its own canonical FQDN to VENs during pairing. This parameter must be set on all nodes in each PCE of the Supercluster.

For example:

supercluster: 
   node_public_ip: 192.168.33.10 
   fqdn: global-pce.mycompany.com
Install Leader

The first step to deploy a new Supercluster is to install and configure the leader PCE, just as you would install a standalone PCE.

For detailed information about installing a PCE, see the PCE Installation and Upgrade Guide.

Install Members

Install each member of your Supercluster by following the exact same procedures you use installing a standalone PCE, except do not create a domain during deployment.

For information about installing a PCE, see the PCE Installation and Upgrade Guide.

Initialize Supercluster Leader

After the leader has been installed, configured, and verified, you initialize the leader.

Note

You must initialize the leader before you start joining any members.

  1. On any node, bring all nodes to runlevel 2:

    sudo -u ilo-pce illumio-pce-ctl set-runlevel 2

    Setting the run level might take some time to complete.

  2. Check the progress with illumio-pce-ctl cluster-status -w to see when the status is Running:

    sudo -u ilo-pce illumio-pce-ctl cluster-status -w

    The nodes must be at runlevel 2 before you run the next command. When all of the nodes have reached runlevel 2, you see the following output:

    Illumio Runtime System                           RUNNING [2] 34.28s

  3. On any node, initialize the leader:

    sudo -u ilo-pce illumio-pce-ctl supercluster-init-leader
Join Each Member to Supercluster

Important

You must join only one member one at a time, and complete all steps before joining the next member. Ensure that each member is at runlevel 2 before joining.

In this procedure, you join the new member to the Supercluster.

All nodes must start at runlevel 2. The nodes should already be at runlevel 2 from the previous procedure.

  1. If necessary, on any node, bring all nodes to runlevel 2:

    sudo -u ilo-pce illumio-pce-ctl set-runlevel 2

  2. On any node, run the following command while you wait for all nodes to reach runlevel 2:

    sudo -u ilo-pce illumio-pce-ctl status --wait

  3. On any core node or the data0 node of the member cluster, join the member to the Supercluster (identified by the leader's FQDN):

    sudo -u ilo-pce illumio-pce-ctl supercluster-join leader_pce_fqdn

    While this command runs, the PCE temporarily sets the runlevel to 1. When the command is interrupted, you might unexpectedly see runlevel 1 .

    Important

    Running this command can take an hour or more depending on the number of PCEs in your Supercluster and size of the PCE database. When this command fails due to network latency, do not proceed until you can run the command again and it executes successfully.

  4. Repeat step 3 for all members you want to join to the Supercluster.

  5. On all PCEs, restart the PCEs in the Supercluster:

    sudo -u ilo-pce illumio-pce-ctl cluster-restart

  6. On all PCEs, bring the PCEs to runlevel 5:

    sudo -u ilo-pce illumio-pce-ctl set-runlevel 5
Verify Supercluster Readiness

Before you begin using your Supercluster, verify that the leader and members are all joined and all PCEs in the Supercluster have a good health status.

Note

It can take up to 10 minutes for all PCEs in your Supercluster to achieve full healthy status.

To verify that your Supercluster is ready to use:

  1. Log into the leader.

  2. On any core node, show Supercluster membership:

    sudo -u ilo-pce illumio-pce-ctl supercluster-members

    The output should show all PCEs in your Supercluster.

  3. Log into the PCE web console of the leader.

  4. Click the Health status icon at the top of the PCE web console. You should see all PCEs in your Supercluster with Normal health status.

If a new PCE being added to the Supercluster has a different value for the parameter service_discovery_encryption_key defined in its runtime_env.yml file than the value specified in the runtime_env.yml files in all the other PCEs in the Supercluster, the new PCE will fail to join the Supercluster.

To remedy this possible problem when a new PCE does not join the Supercluster, follow these steps:

  1. On the new PCE, edit its runtime_env.yml file so that its value for service_discovery_encryption_key is identical to the value set in the runtime_env.yml files of all other Supercluster nodes.

  2. Reset all nodes:

    sudo -u ilo-pce illumio-pce-ctl reset

  3. On all nodes, start services at runlevel 1:

    sudo -u ilo-pce illumio-pce-ctl start --runlevel 1

    Note: If a node gets stuck in the PARTIAL state, reboot the node.

  4. On any node, set up the database:

    sudo -u ilo-pce illumio-pce-db-management setup

  5. On any node, set runlevel 5:

    sudo -u ilo-pce illumio-pce-ctl set-runlevel 5

If a new PCE being added to the Supercluster has a different value for the parameter service_discovery_encryption_key defined in its runtime_env.yml file than the value specified in the runtime_env.yml files in all the other PCEs in the Supercluster, the new PCE will fail to join the Supercluster.

To remedy this possible problem when a new PCE does not join the Supercluster, follow these steps:

  1. On the new PCE, edit its runtime_env.yml file so that its value for service_discovery_encryption_key is identical to the value set in the runtime_env.yml files of all other Supercluster nodes.

  2. Reset all nodes:

    sudo -u ilo-pce illumio-pce-ctl reset

  3. On all nodes, start services at runlevel 1:

    sudo -u ilo-pce illumio-pce-ctl start --runlevel 1

    Note: If a node gets stuck in the PARTIAL state, reboot the node.

  4. On any node, set up the database:

    sudo -u ilo-pce illumio-pce-db-management setup

  5. On any node, set runlevel 5:

    sudo -u ilo-pce illumio-pce-ctl set-runlevel 5
Expand Standalone PCE to Supercluster

To expand your existing standalone PCE to a Supercluster, the steps are similar to the steps for installing a new Supercluster, with additional checks of the standalone PCE before the expansion.

The general workflow for expanding an existing PCE deployment into a Supercluster follows these steps:

  1. Change the pce_fqdn on your standalone PCE; then log into the standalone PCE's web console to verify that the standalone PCE is healthy and is working correctly. See Change FQDN and Verify PCE Health for information.

  2. Verify network connectivity to the database nodes. See Network Connectivity from PCEs to Data Nodes for information.

  3. Initialize your existing PCE as the Supercluster leader. See Initialize Supercluster Leader for information.

  4. Install and configure the new PCEs that will become members of the new Supercluster. See the PCE Installation and Upgrade Guide for information.

  5. Join members to the Supercluster. See Join Each Member to Supercluster for information.

Illumio recommends that you perform each these operations during different change windows.

After your Supercluster is operational, you can reassign workloads connected to the leader to a different PCE in the Supercluster.

Change FQDN and Verify PCE Health

See How to Rename the PCE in the Illumio Knowledge Base for information. (Login required)

Warning

After changing the pce_fqdn parameter and before preceding with the expansion, you must log into the standalone PCE's web console to verify that the standalone PCE is healthy and is working correctly.

Network Connectivity from PCEs to Data Nodes

Before expansion of a Supercluster, ensure that every data node in the standalone cluster can connect to the database nodes via the Supercluster FQDN.

To verify the connections, use telnet or the nc (netcat) utility, which is part of the NMAP set of tools. If not already installed, install NMAP with the following command:

# yum install nmap

Important

Required runlevels

Be sure that the PCEs are set to the following runlevels before checking connectivity:

  • On the PCE from which you run the check: Runlevel 2

  • On the PCEs in other regions that you are checking: Runlevel 2 or higher

For example, you have three regions. With the following nc commands on data0 and data1 in each region, test the connection to the other regions by connecting to port 5432 for the other regions' data0 and data1 nodes.

  • From region 1: Set the PCE from which you are testing to runlevel 2:

    nc -zv region2_data0_ip 5432
nc -zv region3_data0_ip 5432

  • From region 2: Set the PCE from which you are testing to runlevel 2:

    nc -zv region1_data0_ip 5432
nc -zv region3_data0_ip 5432

  • From region 3: Set the PCE from which you are testing to runlevel 2:

    nc -zv region1_data0_ip 5432
nc -zv region2_data0_ip 5432
Migrate to New Supercluster

When you need to migrate your existing Supercluster to a new set of machines, follow these general steps:

  1. On the PCE that is being migrated, pre-configure the IP addresses of the new Supercluster in the runtime_env.yml file. See Pre-configure New IP Addresses for information.

  2. Back up the current Supercluster. See Back Up Supercluster for information.

  3. Restore the old Supercluster configuration and data to the new systems. If you are migrating all PCEs in the Supercluster, see Restore an Entire Supercluster. If you are migrating just one PCE in a Supercluster, see Restore a Single PCE in a Supercluster.

Pre-configure New IP Addresses

Before the migration: When you use DNS-based load balancing (such as round-robin DNS) and are using new IP addresses for the restored PCE, be sure to record those new IP addresses in the runtime_env.yml file on all Supercluster core nodes. This allows VENs to continue to communicate with the PCEs after migration.

Note

When you use traffic-based load balancing, such as with the F5, you do not need to add the new IP addresses to runtime_env.yml. The VENs communicate exclusively with the traffic load balancers' virtual IP addresses, and not directly with the PCEs.

To update runtime_env.yml with additional IP addresses:

If you are migrating more than one PCE, repeat the following steps for each PCE one at a time.

  1. On all nodes of the PCE being migrated, edit the runtime_env.yml file. Under the cluster_public_ips.cluster_fqdn parameter, add the new IP addresses of all of the core nodes in this PCE:

    cluster_public_ips:
  cluster_fqdn:
    - <old IP address>
    - <old IP address>
    - <new IP address>
    - <new IP address>
  cluster_event_service_fqdn:
    - <old IP address>
    - <old IP address>
    - <new IP address>
    - <new IP address>

  2. Restart the PCE that was migrated to send the configuration update to the other PCEs:

    sudo -u ilo-pce illumio-pce-ctl restart

  3. Wait for all services to start on the restarted PCE.

    sudo -u ilo-pce illumio-pce-ctl cluster-status

    The command output should show RUNNING.

  4. Restart the other PCEs in the Supercluster using the same commands in steps 2 and 3.

  5. If you are migrating more than one PCE in the Supercluster, repeat these steps for each PCE one at a time.