Skip to main content

Illumio Core for Kubernetes

Aggregating Logs from Kubelink and C-VEN Pods

There are many log aggregation solutions; this topic describes one example of using Fluent Bit to aggregate our logs. Fluent Bit is a lightweight version of Fluentd with many outputs. See https://docs.fluentbit.io/manual/pipeline/outputs for official details about supported Fluent Bit output plugins.

Loki is used as storage in this example. Change the output section of your Fluent Bit yaml file to suit your needs.

Loki and Grafana

As an example installation for testing, Loki and Grafana are installed in the illumio-system namespace. Loki is installed in monolithic mode to use file system storage For more details, see https://grafana.com/docs/loki/latest/setup/install/helm/install-monolithic/.

helm repo add grafana https://grafana.github.io/helm-charts
helm repo update
helm upgrade --install loki grafana/loki --values loki-values.yaml 
-n illumio-system

Example contents of loki-values.yaml:

loki:
  commonConfig:
    replication_factor: 1
  storage:
    type: 'filesystem'
  auth_enabled: false

singleBinary:
  replicas: 1

# lokiCanary:
#   enabled: false
# gateway:
#   enabled: false
# grafanaAgent:
#   installOperator: true
helm upgrade --install --wait -n illumio-system --set admin.username=admin 
--set admin.password=UseYourPassword --set persistence.enabled=false 
-f grafana-values.yaml grafana 
oci://registry-1.docker.io/bitnamicharts/grafana
kubectl -n illumio-system expose deployment grafana --type=NodePort 
--name=grafana-service
kubectl -n illumio-system get svc grafana-service 
-o go-template='{{range.spec.ports}}
{{if .nodePort}}{{.nodePort}}{{"\n"}}{{end}}{{end}}'

Example contents of grafana-values.yaml:

dashboardsProvider:
  enabled: true
Fluent Bit

The following procedure shows one way of downloading and installing Fluent Bit:

helm repo add fluent https://fluent.github.io/helm-charts 
helm repo update 
helm upgrade --install fluent-bit fluent/fluent-bit --version 0.40.0 
--values fluentbit-values.yaml 
-n illumio-system
kubectl --namespace illumio-system patch daemonsets.apps fluent-bit --patch-file 
fluentbit-patch-nodename.yaml

Example contents of fluentbit-values.yaml: 

labels
  app: IllumioFluentBit

image:
  pullPolicy: IfNotPresent

extraVolumes:
  - name: illumio-ven-data
    hostPath:
    path: /opt/illumio_ven_data
    type: Directory

extraVolumeMounts:
  - name: illumio-ven-data
    mountPath: /opt/illumio_ven_data

config:
  service: |
    [SERVICE]
        daemon Off
        flush {{ .Values.flush }}
        log_level debug
        parsers_file parsers.conf
        parsers_file custom_parsers.conf
        http_server On
        http_listen 0.0.0.0
        http_port {{ .Values.metricsPort }}
        health_check On
  
  inputs: |
    [INPUT]
        Name tail
        Path /var/log/containers/illumio-kubelink*.log
        Tag kubelink.*
        Multiline.parser docker,cri
        Read_From_Head true
        Buffer_Chunk_Size 3MB
        Buffer_Max_Size 10MB
        Mem_Buf_Limit 10MB
        Skip_Long_Lines Off
    [INPUT]
        Name tail
        Path /opt/illumio_ven_data/log/*.log
        Tag cven.*
        Read_From_Head true
        Buffer_Chunk_Size 3MB
        Buffer_Max_Size 10MB
        Mem_Buf_Limit 10MB
        Skip_Long_Lines Off
   
  filters: |
    [FILTER]
        Name kubernetes
        Match kubelink.*
        Merge_Log On
        Kube_Tag_Prefix  kubelink.var.log.containers.
        Merge_Log_Key log_processed
    [FILTER]
        Name parser
        Parser cvenparser
        Match cven.*
        Key_name log
        Preserve_key false
        Reserve_data true
    [FILTER]
        Name record_modifier
        Match cven.*
        Record nodename ${K8S_NODE_NAME}
    
  upstream: {}
				
  outputs: |
    [OUTPUT]
        #for debugging only should be turned off in PROD
        #PLEASE TURN OFF IN PROD		
        Name stdout
        Match *
				
    [OUTPUT]
        Name                   loki
        Match                  *
        Host                   loki.illumio-system.svc.cluster.local
        Port                   3100
        Labels                 job=fluentbit

  customParsers: |
    [PARSER]
      Name        cvenparser
      Format      regex
      Regex       ^(?<time>[^ ]+) (?<message>.+)$
      Time_Key	   time
      Time_Format %Y-%m-%dT%H:%M:%S.%L

  extraFiles {}
				
logLevel: info

Example contents of fluentbit-patch-nodeport.yaml:

spec:
  template:
      spec:
      containers:
      - name: fluent-bit
        env:
          - name: K8S_NODE_NAME
            valueFrom:
              fieldRef:
                fieldPath: spec.nodeName