Kubelink Monitoring and Troubleshooting

If you deployed Illumio Core for Kubernetes 3.0.0 or later, Kubelink is deployed as part of the overall Helm Chart deployment, as described in Deployment with Helm Chart (Core for Kubernetes 3.0.0 and Higher).

Kubelink Process

Kubelink uses a single Ruby process which runs as: ruby /illumio/init.rb.

Kubelink Startup Log Messages

After deploying Kubelink (whether by Helm Chart or manually), verify your deployment with the kubectl get pods -n illumio-system command. The kubelinkpod should be shown with the Running status. In addition, you can review the log file entries after the deployment with the kubectl logs command pointing to the Kubelink pod name.

kubectl logs <kubelink_pod_name> -n illumio-system

A typical successful Kubelink deployment produces log entries similar to these:

I, [2022-05-23T14:36:53.847248 #10]  INFO -- : Starting Kubelink for PCE https://192.168.88.127:10443
I, [2022-05-23T14:36:53.847502 #10]  INFO -- : Metrics reporting enabled; reporting window 30
I, [2022-05-23T14:36:53.847520 #10]  INFO -- : PCE fqdn https://192.168.88.127:10443
[WARNING; em-http-request] TLS hostname validation is disabled (use 'tls: {verify_peer: true}'), see CVE-2020-13482 and https://github.com/igrigorik/em-http-request/issues/339 for details
I, [2022-05-23T14:36:53.893048 #10]  INFO -- : Successfully connected to PCE
I, [2022-05-23T14:36:53.893170 #10]  INFO -- : begin sync on resource namespaces
[WARNING; em-http-request] TLS hostname validation is disabled (use 'tls: {verify_peer: true}'), see CVE-2020-13482 and https://github.com/igrigorik/em-http-request/issues/339 for details
I, [2022-05-23T14:36:53.904369 #10]  INFO -- : Synchronized 6 namespaces.
I, [2022-05-23T14:36:53.904424 #10]  INFO -- : sync on resource namespaces successful, setting up resource version to 184232
I, [2022-05-23T14:36:53.904522 #10]  INFO -- : Start watch on namespaces with version 184232
I, [2022-05-23T14:36:53.905678 #10]  INFO -- : begin sync on resource nodes
[WARNING; em-http-request] TLS hostname validation is disabled (use 'tls: {verify_peer: true}'), see CVE-2020-13482 and https://github.com/igrigorik/em-http-request/issues/339 for details
[WARNING; em-http-request] TLS hostname validation is disabled (use 'tls: {verify_peer: true}'), see CVE-2020-13482 and https://github.com/igrigorik/em-http-request/issues/339 for details
I, [2022-05-23T14:36:53.918093 #10]  INFO -- : Synchronized 1 nodes.
I, [2022-05-23T14:36:53.918143 #10]  INFO -- : sync on resource nodes successful, setting up resource version to 184232
I, [2022-05-23T14:36:53.918175 #10]  INFO -- : Start watch on nodes with version 184232
I, [2022-05-23T14:36:53.919265 #10]  INFO -- : begin sync on resource pods
I, [2022-05-23T14:36:53.935536 #10]  INFO -- : sync on resource pods successful, setting up resource version to 184232
I, [2022-05-23T14:36:53.935601 #10]  INFO -- : Start watch on pods with version 184232
I, [2022-05-23T14:36:53.936938 #10]  INFO -- : begin sync on resource services
[WARNING; em-http-request] TLS hostname validation is disabled (use 'tls: {verify_peer: true}'), see CVE-2020-13482 and https://github.com/igrigorik/em-http-request/issues/339 for details
[WARNING; em-http-request] TLS hostname validation is disabled (use 'tls: {verify_peer: true}'), see CVE-2020-13482 and https://github.com/igrigorik/em-http-request/issues/339 for details
[WARNING; em-http-request] TLS hostname validation is disabled (use 'tls: {verify_peer: true}'), see CVE-2020-13482 and https://github.com/igrigorik/em-http-request/issues/339 for details
I, [2022-05-23T14:36:54.029965 #10]  INFO -- : Synchronized 3 services, full=true, force=false
I, [2022-05-23T14:36:54.030013 #10]  INFO -- : sync on resource services successful, setting up resource version to 184232
I, [2022-05-23T14:36:54.030046 #10]  INFO -- : Start watch on services with version 184232
I, [2022-05-23T14:36:54.031042 #10]  INFO -- : begin sync on resource replica_sets
I, [2022-05-23T14:36:54.100090 #10]  INFO -- : Nothing to sync
I, [2022-05-23T14:36:54.100237 #10]  INFO -- : sync on resource replica_sets successful, setting up resource version to 184232
I, [2022-05-23T14:36:54.100281 #10]  INFO -- : Start watch on replica_sets with version 184232
I, [2022-05-23T14:36:54.101226 #10]  INFO -- : begin sync on resource stateful_sets
I, [2022-05-23T14:36:54.170175 #10]  INFO -- : Nothing to sync
I, [2022-05-23T14:36:54.170220 #10]  INFO -- : sync on resource stateful_sets successful, setting up resource version to 184232
I, [2022-05-23T14:36:54.170267 #10]  INFO -- : Start watch on stateful_sets with version 184232
I, [2022-05-23T14:36:54.171159 #10]  INFO -- : begin sync on resource daemon_sets
I, [2022-05-23T14:36:54.245866 #10]  INFO -- : Nothing to sync
I, [2022-05-23T14:36:54.246025 #10]  INFO -- : sync on resource daemon_sets successful, setting up resource version to 184232
I, [2022-05-23T14:36:54.246210 #10]  INFO -- : Start watch on daemon_sets with version 184232
I, [2022-05-23T14:36:54.247946 #10]  INFO -- : begin sync on resource replication_controllers
I, [2022-05-23T14:36:54.324925 #10]  INFO -- : Nothing to sync
I, [2022-05-23T14:36:54.324977 #10]  INFO -- : sync on resource replication_controllers successful, setting up resource version to 184232
I, [2022-05-23T14:36:54.325032 #10]  INFO -- : Start watch on replication_controllers with version 184232
[WARNING; em-http-request] TLS hostname validation is disabled (use 'tls: {verify_peer: true}'), see CVE-2020-13482 and https://github.com/igrigorik/em-http-request/issues/339 for details
[WARNING; em-http-request] TLS hostname validation is disabled (use 'tls: {verify_peer: true}'), see CVE-2020-13482 and https://github.com/igrigorik/em-http-request/issues/339 for details
[WARNING; em-http-request] TLS hostname validation is disabled (use 'tls: {verify_peer: true}'), see CVE-2020-13482 and https://github.com/igrigorik/em-http-request/issues/339 for details
[WARNING; em-http-request] TLS hostname validation is disabled (use 'tls: {verify_peer: true}'), see CVE-2020-13482 and https://github.com/igrigorik/em-http-request/issues/339 for details
[WARNING; em-http-request] TLS hostname validation is disabled (use 'tls: {verify_peer: true}'), see CVE-2020-13482 and https://github.com/igrigorik/em-http-request/issues/339 for details
I, [2022-05-23T14:36:54.505403 #10]  INFO -- : replica_sets MODIFIED
I, [2022-05-23T14:37:24.312086 #10]  INFO -- : Heart beating to PCE
I, [2022-05-23T14:37:24.312191 #10]  INFO -- : Attaching metrics report to heartbeat: {:pod_changes=>[{:namespace=>"illumio-system", "added"=>0, "modified"=>0, "deleted"=>1}], :service_changes=>[], :duration_seconds=>30}
[WARNING; em-http-request] TLS hostname validation is disabled (use 'tls: {verify_peer: true}'), see CVE-2020-13482 and https://github.com/igrigorik/em-http-request/issues/339 for details
I, [2022-05-23T14:37:54.343467 #10]  INFO -- : Heart beating to PCE
I, [2022-05-23T14:37:54.343874 #10]  INFO -- : Attaching metrics report to heartbeat: {:pod_changes=>[], :service_changes=>[], :duration_seconds=>30}
[WARNING; em-http-request] TLS hostname validation is disabled (use 'tls: {verify_peer: true}'), see CVE-2020-13482 and https://github.com/igrigorik/em-http-request/issues/339 for details
I, [2022-05-23T14:38:24.373847 #10]  INFO -- : Heart beating to PCE
I, [2022-05-23T14:38:24.373924 #10]  INFO -- : Attaching metrics report to heartbeat: {:pod_changes=>[], :service_changes=>[], :duration_seconds=>30}
[WARNING; em-http-request] TLS hostname validation is disabled (use 'tls: {verify_peer: true}'), see CVE-2020-13482 and https://github.com/igrigorik/em-http-request/issues/339 for details
I, [2022-05-23T14:38:54.380933 #10]  INFO -- : Heart beating to PCE
I, [2022-05-23T14:38:54.381009 #10]  INFO -- : Attaching metrics report to heartbeat: {:pod_changes=>[], :service_changes=>[], :duration_seconds=>30}
[WARNING; em-http-request] TLS hostname validation is disabled (use 'tls: {verify_peer: true}'), see CVE-2020-13482 and https://github.com/igrigorik/em-http-request/issues/339 for details
I, [2022-05-23T14:39:24.401636 #10]  INFO -- : Heart beating to PCE
I, [2022-05-23T14:39:24.401748 #10]  INFO -- : Attaching metrics report to heartbeat: {:pod_changes=>[], :service_changes=>[], :duration_seconds=>30}
[WARNING; em-http-request] TLS hostname validation is disabled (use 'tls: {verify_peer: true}'), see CVE-2020-13482 and https://github.com/igrigorik/em-http-request/issues/339 for details
I, [2022-05-23T14:39:54.422494 #10]  INFO -- : Heart beating to PCE
I, [2022-05-23T14:39:54.422595 #10]  INFO -- : Attaching metrics report to heartbeat: {:pod_changes=>[], :service_changes=>[], :duration_seconds=>30}
[WARNING; em-http-request] TLS hostname validation is disabled (use 'tls: {verify_peer: true}'), see CVE-2020-13482 and https://github.com/igrigorik/em-http-request/issues/339 for details
I, [2022-05-23T14:40:24.453077 #10]  INFO -- : Heart beating to PCE
I, [2022-05-23T14:40:24.453217 #10]  INFO -- : Attaching metrics report to heartbeat: {:pod_changes=>[], :service_changes=>[], :duration_seconds=>30}
[WARNING; em-http-request] TLS hostname validation is disabled (use 'tls: {verify_peer: true}'), see CVE-2020-13482 and https://github.com/igrigorik/em-http-request/issues/339 for details
I, [2022-05-23T14:40:54.466210 #10]  INFO -- : Heart beating to PCE
I, [2022-05-23T14:40:54.466455 #10]  INFO -- : Attaching metrics report to heartbeat: {:pod_changes=>[], :service_changes=>[], :duration_seconds=>30}
[WARNING; em-http-request] TLS hostname validation is disabled (use 'tls: {verify_peer: true}'), see CVE-2020-13482 and https://github.com/igrigorik/em-http-request/issues/339 for details
I, [2022-05-23T14:41:24.296410 #10]  INFO -- : Verify watches for ["namespaces", "nodes", "pods", "services", "replica_sets", "stateful_sets", "daemon_sets", "replication_controllers"]
I, [2022-05-23T14:41:24.296468 #10]  INFO -- : Watch client namespaces Connection Idle: 270.3355407714844s
I, [2022-05-23T14:41:24.296485 #10]  INFO -- : Watch client nodes Connection Idle: 179.93679809570312s
I, [2022-05-23T14:41:24.296499 #10]  INFO -- : Watch client pods Connection Idle: 240.5237274169922s
I, [2022-05-23T14:41:24.296513 #10]  INFO -- : Watch client services Connection Idle: 270.0260314941406s
I, [2022-05-23T14:41:24.296526 #10]  INFO -- : Watch client replica_sets Connection Idle: 269.85888671875s
I, [2022-05-23T14:41:24.296542 #10]  INFO -- : Watch client stateful_sets Connection Idle: 270.0269775390625s
I, [2022-05-23T14:41:24.296573 #10]  INFO -- : Watch client daemon_sets Connection Idle: 270.02490234375s
I, [2022-05-23T14:41:24.296731 #10]  INFO -- : Watch client replication_controllers Connection Idle: 270.02490234375s
[WARNING; em-http-request] TLS hostname validation is disabled (use 'tls: {verify_peer: true}'), see CVE-2020-13482 and https://github.com/igrigorik/em-http-request/issues/339 for details
I, [2022-05-23T14:41:24.300532 #10]  INFO -- : Synchronized 3 services, full=true, force=true
I, [2022-05-23T14:41:24.452846 #10]  INFO -- : Heart beating to PCE
[WARNING; em-http-request] TLS hostname validation is disabled (use 'tls: {verify_peer: true}'), see CVE-2020-13482 and https://github.com/igrigorik/em-http-request/issues/339 for details
W, [2022-05-23T14:41:54.186807 #10]  WARN -- : watch client for stateful_sets error callback invoked. Resetting watch ...
W, [2022-05-23T14:41:54.186863 #10]  WARN -- : Watch on stateful_sets ended. Resetting it after 3 seconds
I, [2022-05-23T14:41:54.441880 #10]  INFO -- : Heart beating to PCE
I, [2022-05-23T14:41:54.441991 #10]  INFO -- : Attaching metrics report to heartbeat: {:pod_changes=>[], :service_changes=>[], :duration_seconds=>60}
[WARNING; em-http-request] TLS hostname validation is disabled (use 'tls: {verify_peer: true}'), see CVE-2020-13482 and https://github.com/igrigorik/em-http-request/issues/339 for details
I, [2022-05-23T14:41:57.193339 #10]  INFO -- : begin sync on resource stateful_sets
I, [2022-05-23T14:41:57.267375 #10]  INFO -- : Nothing to sync
I, [2022-05-23T14:41:57.267411 #10]  INFO -- : sync on resource stateful_sets successful, setting up resource version to 184451
I, [2022-05-23T14:41:57.267424 #10]  INFO -- : Start watch on stateful_sets with version 184451
[WARNING; em-http-request] TLS hostname validation is disabled (use 'tls: {verify_peer: true}'), see CVE-2020-13482 and https://github.com/igrigorik/em-http-request/issues/339 for details
I, [2022-05-23T14:42:24.483142 #10]  INFO -- : Heart beating to PCE
I, [2022-05-23T14:42:24.483224 #10]  INFO -- : Attaching metrics report to heartbeat: {:pod_changes=>[], :service_changes=>[], :duration_seconds=>30}
[WARNING; em-http-request] TLS hostname validation is disabled (use 'tls: {verify_peer: true}'), see CVE-2020-13482 and https://github.com/igrigorik/em-http-request/issues/339 for details

Verify Kubelink Deployment

To verify your Kubelink deployment.

To check the Kubelink Pod status for Kubernetes:
```
kubectl get pods -n illumio-system
```
To check the Kubelink Pod status for OpenShift:
```
oc get pods -n illumio-system
```

The illumio-kubelink-xxxxxxxxxx-xxxxx Pod should be in the "Running" state. If the either get pods -n illumio-system command shows the kubelink pod is not successfully running, check the log file for any ERROR messages.

After Kubelink is successfully deployed, you can check the cluster information in the Illumio PCE UI. From the main menu, navigate to Infrastructure > Container Clusters.

Below is an example of a healthy container cluster state reported by Kubelink, where Status is "In Sync".

You can also verify in the PCE UI that Kubelink was successfully deployed by checking the following:

Under the Container Workload Profiles tab, namespaces created in your Kubernetes or OpenShift cluster should be listed. An example is shown below.
Under Policy Objects > Virtual Services, services created in your Kubernetes or OpenShift cluster should be listed. An example is shown below.

PCE-Kubelink Connection and Heartbeat

The Kubelink heartbeat to the PCE is logged in its log file. Use the kubectl logs command, and search for the Heart beating to PCE string to confirm. To confirm PCE-Kubelink connectivity, check the PCE UI, which will show the Kubelink pod as being offline if the heartbeat is missing 2-3 times (about 10 minutes).

Additional Kubelink Monitoring

Other Kubelink actions that can be confirmed in the Kubelink log file include:

API request succeeds

When Kubelink successfully sets up a watch with the Kubernetes API, the related log entry is:

sync on resource <RESOURCE> successful, setting up resource version to <RESOURCE VERSION>

Information sent to PCE

When Kubelink successfully sends information to the PCE, the related log entry is:

Synchronized 2 <RESOURCE>, full=..., force=...

Setting Log Verbosity

The log verbosity level is set by default to include INFO, WARNING, and ERROR messages in the log. If your log appears to be extremely small (showing only ERRORs, for example), or is extremely large (which could indicate being set at the DEBUG level), you can check the log_level setting in the illumio-kubelink-secret.yml file. Values for this setting are:

log_level Setting	Description
0	Debug
1	Info (default)
2	Warn
3	Error

Values are cumulative, meaning that a setting includes all other settings greater than it. For example, the default setting of '1' includes all INFO, WARNING, and ERROR messages in the log file, but a setting of '3' would only include ERROR messages.

Illumio Install, Configure, and Upgrade Guide 24.2.20