Skip to main content

Illumio Install, Configure, and Upgrade Guide 24.2.20

Deploy a PCE Supercluster

You can deploy the Illumio Supercluster in several ways:

  • New: You have never deployed a PCE and want to deploy a new Supercluster.

    See Deploy a New Supercluster.

  • Expand: You have already deployed a standalone PCE and want to expand it to a Supercluster. See Expand Standalone PCE to Supercluster.

  • Join: You already have more than one standalone PCE and want to combine them into a Supercluster. Contact Illumio Customer Support for assistance.

Deploy a New Supercluster

  1. Install the leader PCE as a standalone PCE.

  2. Install and configure each member PCE as a standalone PCE.

  3. Initialize the Supercluster leader.

  4. Join members to the Supercluster.

  5. Bring the leader and members to a fully operational state.

  6. Verify that the Supercluster is ready for use.

Note

Any time requirements do not bind the sequence of events for deploying a Supercluster; for example, there is no time limit between initializing a Supercluster leader and joining individual members.

Before You Begin: Runtime Configuration

Before you deploy your PCE Supercluster, be aware of the following runtime_env.yml configurations:

  • The value of the parameter service_discovery_encryption_key in the runtime_env.yml file must be the same on all nodes on all PCEs in your Supercluster.

  • You do not need to configure the public IP addresses of other PCEs under the cluster_public_ips parameter. Supercluster PCEs automatically exchange their configured public IP addresses with each other, which the VEN programmed to allow workloads to migrate between PCEs.

Optional

Depending on your deployment environment, you might need to make the following changes to the runtime_env.yml file on each PCE in the Supercluster.

When the nodes of each PCE use multiple IP addresses or they use IP addresses other than the one advertised on the node for communication with other PCEs, such as having a NAT between the PCEs in your Supercluster, configure this optional parameter:

  • supercluster.node_public_ip: The public IP address of this node is advertised to other PCEs in your Supercluster deployment. This IP address must be accessible from all other Supercluster PCEs you want to join. This parameter must be set on all nodes in each PCE. When your PCE is deployed in a public cloud, such as AWS, this must be a public IP address.

Install Leader

The first step to deploy a new Supercluster is to install and configure the leader PCE, just as you would install a standalone PCE.

For detailed information about installing a PCE, see thePCE Installation and Upgrade Guide .

Install Members

Install each Supercluster member by following the same procedures you use to install a standalone PCE, except do not create a domain during deployment.

For information about installing a PCE, see the PCE Installation and Upgrade Guide.

Initialize Supercluster Leader

After the leader has been installed, configured, and verified, you initialize the leader.

Note

You must initialize the leader before you start joining any members.

  1. On any node, bring all nodes to runlevel 2:

    sudo -u ilo-pce illumio-pce-ctl set-runlevel 2

    Setting the run level might take some time to complete.

  2. Check the progress with illumio-pce-ctl cluster-status -w to see when the status is Running:

    sudo -u ilo-pce illumio-pce-ctl cluster-status -w

    The nodes must be at runlevel 2 before you run the next command. When all of the nodes have reached runlevel 2, you see the following output:

    Illumio Runtime System                           RUNNING [2] 34.28s

  3. On any node, initialize the leader:

    sudo -u ilo-pce illumio-pce-ctl supercluster-init-leader
Join Each Member to Supercluster

Important

You must join only one member at a time, and complete all steps before joining the next member. Ensure that each member is at runlevel 2 before joining.

Join the new member to the Supercluster.

All nodes must start at runlevel 2. The nodes should already be at runlevel 2 from the previous procedure.

  1. If necessary, on any node, bring all nodes to runlevel 2:

    sudo -u ilo-pce illumio-pce-ctl set-runlevel 2

  2. On any node, run the following command while you wait for all nodes to reach runlevel 2:

    sudo -u ilo-pce illumio-pce-ctl cluster-status --wait

  3. On any core node or the data0 node of the member cluster, join the member to the Supercluster (identified by the leader's FQDN):

    sudo -u ilo-pce illumio-pce-ctl supercluster-join leader_pce_fqdn

    While this command runs, the PCE temporarily sets the runlevel to 1. When the command is interrupted, you might unexpectedly see runlevel 1 .

    Important

    Running this command can take an hour or more, depending on the number of PCEs in your Supercluster and the size of the PCE database. When this command fails due to network latency, do not proceed until you can run it again and it executes successfully.

  4. Repeat step 3 for all members you want to join the Supercluster.

  5. On all PCEs, restart the PCEs in the Supercluster:

    sudo -u ilo-pce illumio-pce-ctl cluster-restart

  6. On all PCEs, bring the PCEs to runlevel 5:

    sudo -u ilo-pce illumio-pce-ctl set-runlevel 5
Verify Supercluster Readiness

Before you begin using your Supercluster, verify that the leader and members are all joined and all PCEs in the Supercluster have a good health status.

Note

It can take up to 10 minutes for all PCEs in your Supercluster to achieve full healthy status.

To verify that your Supercluster is ready to use:

  1. Log in to the leader.

  2. On any core node, show Supercluster membership:

    sudo -u ilo-pce illumio-pce-ctl supercluster-members

    The output should show all PCEs in your Supercluster.

  3. Log in to the PCE web console of the leader.

  4. Click the Health status icon at the top of the PCE web console. You should see all PCEs in your Supercluster with Normal health status.

If a new PCE being added to the Supercluster has a different value for the parameter service_discovery_encryption_key defined in its runtime_env.yml file than the value specified in the runtime_env.yml files in all the other PCEs in the Supercluster, the new PCE will fail to join the Supercluster.

To remedy this possible problem when a new PCE does not join the Supercluster, follow these steps:

  1. On the new PCE, edit its runtime_env.yml file so that its value for service_discovery_encryption_key is identical to the value set in the runtime_env.yml files of all other Supercluster nodes.

  2. Reset all nodes:

    sudo -u ilo-pce illumio-pce-ctl reset

  3. On all nodes, start services at runlevel 1:

    sudo -u ilo-pce illumio-pce-ctl start --runlevel 1

    Note: If a node gets stuck in the PARTIAL state, reboot the node.

  4. On any node, set up the database:

    sudo -u ilo-pce illumio-pce-db-management setup

  5. On any node, set runlevel 5:

    sudo -u ilo-pce illumio-pce-ctl set-runlevel 5