Expand 2x2 Cluster to 4x2

This section describes how to expand an existing PCE 2x2 cluster to a 4x2 cluster by adding two core nodes.

Prepare Environment for Cluster Expansion

This section helps you prepare your PCE cluster environment for the new core nodes.

Prepare Server Load Balancer or DNS

Add the new core node information for a server load balancer (SLB) or DNS:

Server load balancer (SLB)
Before installing the PCE software on the two new core nodes, perform the following tasks:
- Add the IP addresses of the two new nodes to your load balancer configuration.
- Configure your load balancer to check the health of the new core nodes.
- Run a health check and verify that the two new core nodes are down.
- Verify that traffic is not being forwarded to the new nodes.
DNS
Perform the following tasks:
- Add the two new nodes to your DNS configuration.
- When TCP connectivity from the VENs to the PCE is direct and not routed through a virtual IP (VIP), modify the runtime_env.yml on all four nodes in the existing cluster and change the cluster_public_ip > cluster_fqdn to include the two new core nodes.
  Define this parameter as a list of IP addresses that the VENs can connect to, which is the load balancing VIP or a list of all core nodes in the cluster.
  For example:
```
cluster_public_ips:
    cluster_fqdn:
        - <existing_core_node_ip_address>
        - <existing_core_node_ip_address>
        - <new_core_ip_node_address>
        - <new_core_ip_node_address>
```

Ensure Connectivity from VENs to New Nodes

Ensure that connectivity from existing VENs to the new core nodes is allowed and working; for example, you might need to update your network's firewall policies to permit access from existing VENs to the new core nodes.

Prepare the Cluster for New Nodes

Before you install the PCE software on the new core nodes, perform the following tasks.

Stop the cluster by running this command:
```
sudo -u ilo-pce illumio-pce-ctl stop
```
Validate the cluster's configuration by running this command:
```
sudo -u ilo-pce illumio-pce-ctl check-env
```
Start the cluster by running this command:
```
sudo -u ilo-pce illumio-pce-ctl start
```

The PCE configures all VENs to include access to the new core nodes. When complete, all your VENs should be listed as online.

Back Up PCE Database

Before you expand your 2x2 cluster, create a backup of your PCE database.

Configure Existing Nodes for Expansion

On all nodes in the existing cluster, stop the PCE software:
```
sudo -u ilo-pce illumio-pce-ctl stop
```
Before you modify the runtime_env.yml file on the existing nodes, create a file backup in case you need to revert back to the last known configuration.
For example, on all nodes, run this command:
```
cp /etc/illumio-pce/runtime_env.yml /etc/illumio-pce/runtime_env.yml.bak
```
Modify both new core nodes' runtime_env.yml file so that the node_type parameter is defined as core. For example, change the parameter from core0 or core1 to core.
On all nodes, modify the runtime_env.yml file to define the cluster_type parameter as 6node_v0 and save the file. Your runtime_env.yml file might not have this parameter; you only need to add it when it does not already exist.
For example:
```
cluster_type: 6node_v0
```
On all nodes in the existing cluster, check the syntax of the runtime_env.yml configuration:
```
sudo -u ilo-pce illumio-pce-env check
```
On all nodes in the existing cluster, restart the PCE with the configuration changes:
```
sudo -u ilo-pce install_root/illumio-pce-ctl restart
```
On any node in the cluster, check the cluster status:
```
sudo -u ilo-pce install_root/illumio-pce-ctl cluster-status
```
The status of the cluster should return as RUNNING.

Install and Configure PCE on Nodes

Install the PCE software and configure the new core nodes using the same RPM used to install the existing nodes, and use the same system and environmental configuration as the existing two core nodes. This configuration includes all runtime_env.yml settings, kernel performance modifications, syslog configurations, DNS, and NTP.

Caution

Use the same RPM you used to install the existing PCE nodes to install the PCE software on the new nodes.

After you have installed the PCE software, perform these steps:

For layer 4 load balancer implementations, confirm that two of the core nodes are present and UP on the load balancer. These nodes should match with those shown in cluster-status with the role of server_load_balancer. When nodes in the cluster fail, the nodes that own the server_load_balancer role can change.
Ensure that the TLS certificate is valid for the new nodes as well as the existing nodes. The certificate might contain only the cluster name, or might include each of the core node names in the SAN field. When the SAN field is used, ensure that both of the new core nodes are included.
Copy the certificate and key from the existing core nodes to the new core nodes in /var/lib/illumio-pce/cert (or wherever you defined this location in the runtime_env.yml file).
Copy the runtime_env.yml file from an existing core node to the new core nodes. Ensure that when nodes have a specific configuration, such as internal_service_ip, you configure this parameter on the new core nodes to correctly reflect the configuration on the two new nodes.
Verify that the new nodes have the correct node_type (core) and cluster_type (6node_v0) and, when using a DNS load balancer, verify that all four core nodes are defined in the runtime parameter named cluster_public_ips > cluster_fqdn.
On all new core nodes, verify that the new core nodes were configured correctly:
```
sudo -u ilo-pce illumio-pce-ctl check-env
```
Find the IP address of the cluster leader node:
```
sudo -u ilo-pce illumio-pce-ctl cluster-leader
```
On any existing node in the cluster (not the new node you are about to add), run the following command. For ip_address, substitute the IP address of the first new node.
```
sudo -u ilo-pce illumio-pce-ctl cluster-nodes allow ip_address
```
On the first new node, insert the first new core node into the cluster. Use the cluster leader node IP address that you found in the earlier step.
```
sudo -u ilo-pce illumio-pce-ctl cluster-join ip_address_of_leader_node
```
This command should confirm the node is added and report that there are 5 nodes in the cluster.
On any existing node in the cluster (not the second new node you are about to add), run the following command. For ip_address, substitute the IP address of the second new node.
```
sudo -u ilo-pce illumio-pce-ctl cluster-nodes allow ip_address
```
On the second new node, insert the second new core node into the cluster:
```
sudo -u ilo-pce illumio-pce-ctl cluster-join ip_address_of_leader_node
```
This command should confirm the node is added and report that there are 6 nodes in the cluster.
On all nodes, restart the PCE software with the configuration changes:
```
sudo -u ilo-pce illumio-pce-ctl restart
```

Verify Cluster Expansion

Perform these steps to ensure that you have successfully expanded your PCE 2x2 to a 4x2 cluster.

To verify that the cluster is fully up and running and all PCE services are at runlevel 5, run the status command:
```
sudo -u ilo-pce illumio-pce-ctl cluster-status
```

Confirm that the cluster contains 6 nodes:

sudo -u ilo-pce illumio-pce-ctl cluster-members

When you are using a server load balancer to manager PCE traffic, confirm on the load balancer that two of the core nodes are present and listed as UP. These nodes should match those shown from the cluster-status command with the role of server_load_balancer. When nodes in the cluster fail, the nodes that own the server_load_balancer role can change.
Verify that you can log into the PCE web console and navigate the interface successfully.
Verify that logs are being populated in the logging directory of the new nodes, and (when configured) logs are being forwarded to external log destinations.
Verify that your workload VENs are online in the Workloads page of the PCE web console. Be aware that VENs might be offline occasionally for unrelated reasons; therefore, compare the VEN connectivity status to your baseline.
Note
Large numbers of VENs remaining in Syncing state can indicate that one of the core nodes is not reachable due to a network firewall, load balancer, or runtime_env.yml misconfiguration.

Illumio Core 24.5 Administration Guide