Skip to main content

What's New and Release Notes for 26.x

Setting up Illumio VEN on NVIDIA DPU

Follow this guide to set up and run an Illumio VEN on NVIDIA BlueField-3 DPU.

Prerequisites

NVIDIA prerequisites
  • NVIDIA BlueField DPU Image: v4.11.0 or later

  • NVIDIA BlueField DPU Firmware: v24.35.0388 or later

  • NVIDIA BlueField DPU Configuration: Follow NVIDIA docs to configure the DPU in DPU Mode (also known as Embedded CPU Function Ownership (ECPF) mode).

  • DOCA Installation: Follow NVIDIA docs to install DOCA for both host and DPU.

Illumio prerequisites
  • Host/VM OS: Linux

  • PCE: use the PCE provided by Illumio

  • Illumio VEN: use the VEN package provided by Illumio (illumio-ven-25.2.40-4000.u16.arm64.deb)

Set up host

Set up your host for NVIDIA DOCA Argus.

Enable IOMMU passthrough

On your host, edit the GRUB configuration file to enable IOMMU passthrough.

  1. Open the GRUB configuration file:

    sudo vim /etc/default/grub
  2. Add the following to GRUB_CMDLINE_LINUX_DEFAULT:

    iommu=pt <intel/amd>_iommu=on
  3. Save the file and apply the changes.

    For example, on Ubuntu run sudo update-grub.

  4. Reboot your host.

Prepare target system

Tip

Repeat this section once for each distinct target system environment whose memory the DPU will scan on the host system. A distinct environment is defined by a unique operating system and OS version (for example, Ubuntu 20.04 vs. Ubuntu 22.04, or different guest VM operating systems). You only need to repeat this preparation when the OS or OS version differs; otherwise, multiple systems running the same OS version can reuse the same prepared artifacts.

Prepare the target system configuration separately for each unique operating system and OS version the DPU will scan on the host system (for example, each distinct VM OS version running on the host).

Once the memory regions and OS symbols mappings are generated they can be reused for any target system of that OS version. If more than one type of system (VM/host) is being scanned, these regions and mappings may be added to the config folder of subdirectories for each supported system.

Once the memory region definitions and OS symbol mappings are generated, they can be reused for any target system running the same OS and OS version. If the DPU scans more than one type of target system (for example, host and VM), store the generated artifacts in separate subdirectories under the configuration folder, organized by OS and OS version.

  1. Download OS debug symbols.

    1. For Ubuntu:

      sudo tee /etc/apt/sources.list.d/ddebs.list << EOF
      deb http://ddebs.ubuntu.com/ $(lsb_release -cs) main restricted universe multiverse
      deb http://ddebs.ubuntu.com/ $(lsb_release -cs)-updates main restricted universe multiverse
      deb http://ddebs.ubuntu.com/ $(lsb_release -cs)-proposed main restricted universe multiverse
      EOF
      sudo apt install ubuntu-dbgsym-keyring
      sudo apt-get update
      sudo apt-get install linux-image-$(uname -r)-dbgsym
  2. Build dwarf2json:

    1. Clone the dwarf2json repo.

    2. Navigate to the repo and build:

      cd dwarf2json && go build
  3. Create a directory to store target symbols:

    mkdir target_symbols
  4. Create JSON files and copy them to target_symbols :

    cd /opt/mellanox/doca/tools/
    pip3 install psutil pdbparse
    sudo python3 /opt/mellanox/doca/tools/doca_apsh_config.py --files memregions symbols --os <windows/linux> --path <path to dwarf2json>
    cp /opt/mellanox/doca/tools/*.json ~illumio/target_symbols
    
  5. Copy all files from target_symbols to the DPU via scp to a destination directory of your choice (for example, /etc/doca_argus_config/). The copied files should include Profile_*.json and mem_regions.json.

    You will reference the destination directory (for example, /etc/doca_argus_config) during DPU setup.

Configure NVIDIA DOCA Virtual Functions

Follow the NVIDIA guide to configure NVIDIA DOCA Virtual Functions (VFs on DPU, VF representors on host).

Tip

Refer to NVIDIA docs for help configuring SR-IOV virtual functions on the host.

Set up DPU

After preparing a target system, install the VEN on the DPU and run the DOCA Argus container.

Configure BlueField firmware
  1. On the BlueField system, configure the PF base register and NVMe emulation settings:

    mlxconfig -d /dev/mst/mt41686_pciconf0 s PF_BAR2_SIZE=2 PF_BAR2_ENABLE=1 NVME_EMULATION_ENABLE=1
  2. To apply the settings, follow the NVIDIA guide to perform a BlueField system reboot.

    1. After rebooting, you can run a command toverify whether the settings were successfully applied:

      sudo mlxconfig -d /dev/mst/mt41686_pciconf0 q | grep -E "NVME|BAR"

      The output should include the following configurations:

      PF_BAR2_SIZE            2
      PF_BAR2_ENABLE          True(1)
      NVME_EMULATION_ENABLE   True(1)
  3. Configure huge pages for the DPU:

    echo '1024' | sudo tee -a /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages
    sudo mkdir /mnt/huge
    sudo mount -t hugetlbfs -o pagesize=2M nodev /mnt/huge
Configure DOCA Argus

Tip

Refer to DOCA Argus Service Guide.

  1. Modify doca_argus.yaml:

    1. Under volumes, add a volume named conf-argus-config with the path on the DPU at which the doca_apsh_config.py output files were saved to in step 2(d) of the Host Setup:

      volumes:    
          - name: logs-argus-service      
            hostPath:        
              path: /var/log/doca_argus        
              type: DirectoryOrCreate    
          - name: logs-argus-activity-report      
            hostPath:        
              path: /var/log/doca_argus_activity_report        
              type: DirectoryOrCreate    
          - name: conf-argus-config     #<--------------Add this volume      
            hostPath:        
              path: /etc/doca_argus_config  #<-----------Path from step 2(d)        
              type: DirectoryOrCreate
    2. Under volumeMounts, mount a volume named conf-argus-config to a path of your choice:

      volumeMounts:      
          - name: logs-argus-service        
            mountPath: /var/log/doca_argus      
          - name: logs-argus-activity-report        
            mountPath: /var/log/doca_argus_activity_report      
          - name: conf-argus-config        
            mountPath: /etc/doca_argus_config  #<------------ Choose a path
    3. Under apsh_config, set os_symbol_path and men_regions_path to be the same path you chose previously for conf-argus-config:

      apsh_config:  
          # When true, scans for additional systems beyond those specified in config. Always enabled if systems section is empty  
          auto_scan: true  
          # Default values applied to all systems (including auto-scanned systems) unless overridden in the system section  
          default:    
            # OS symbol manifest path - can be single json, folder of jsons or auto. 'auto' means automatic detection    
            os_symbol_path: /etc/doca_argus_config/    
            # Memory regions path. 'auto' means automatic detection    
            mem_regions_path: /etc/doca_argus_config/  #<------------------- Same path as conf-argus-config
  2. Copy doca_argus.yaml to the path required to start the DOCA Argus container:

    cp doca_argus.yaml /etc/kubelet.d/
Configure OVS

Configure an OVS (Open vSwitch) bridge and attach an sFlow config to it to provide visibility to traffic between host and DPU.

  • Add representor devices to an OVS bridge.

    • Choose an OVS bridge name, for example ovsbr2.

    • Choose a sampling rate, for example sampling=512.

    • Choose your representor device(s).

      An example representor device is pf0hpf. If you have representors configured for host-side networking and you want to discover their corresponding workloads, you may configure those as well.  Note: this configuration is functionally creating a bridge between representor devices.

    ovs-vsctl add-br ovsbr2
    ovs-vsctl add-port ovsbr2 <representor-device>
    ovs-vsctl -- --id=@sflow create sflow agent=oob_net0 target=\"127.0.0.1:6343\" sampling=512 polling=10 -- set bridge ovsbr2 sflow=@sflow
Install the VEN

Download and install the VEN:

apt install ./illumio-ven-25.2.40-4000.u16.arm64.deb

Set up the PCE

To enable workload discovery reporting via DOCA Argus, configure the Illumio PCE to use C-VEN (Containerized VEN) APIs.

  1. Pair the VEN with the PCE.

  2. On the PCE, create a container cluster node object.

    Save the cluster UUID, Org ID, and Token (in <org-id>_Token format) for a subsequent step.

  3. Get the machine ID of the DPU where the VEN is deployed:

    cat /etc/machine-id

    Save the machine ID for a subsequent step.

  4. Send a request to the PCE.

    Use these options in the request:

    • cluster_uuid: use your saved cluster UUID

    • org_id: use your saved cluster Org ID

    • cluster_token_value: modify your saved cluster token by removing its org prefix.

      For example, given a cluster token of 2_123456789, remove the org prefix 2_ and use 123456789 for the cluster_token_value.

    curl https://<your pce address>:8443/api/v2/orgs/<org_id>/container_clusters/<cluster_uuid>/put_from_cluster -v -u kubelink_<cluster_uuid>:<cluster_token_value> -X PUT -H "Content-Type: application/json" -d '{"nodes": [{"machine_id": "<your machine id>", "pod_subnet": "172.17.0.0/16"}]}'
  5. If you receive a 429 response code, you may need to increase the PCE object limits on the number of interfaces per agent.

    Log in to the PCE and configure its interfaces per agent:

    /opt/illumio-pce/illumio-pce-ctl obj-limits set --profile Default --key interfaces_per_agent --hard_limit 64
  6. Create a Container Workload Profile on the PCE for the newly created container cluster.

Validate setup

Use Illumio Console to test and validate that your VEN on DPU is activated, workloads are discovered, and traffic is captured and analyzed.

Validate VEN activation

In Illumio Console, validate that VEN activation was successful:

  • VEN status is Active/Connected

  • The DPU host appears with the correct hostname, IP addresses, labels, and interfaces.

Validate workload discovery

In Illumio Console, validate that workloads have been discovered:

  • VMs on the DPU host are inventoried and labeled.

  • Downstream workloads via DPU-routed traffic are visible and discoverable.

Analyze discovered traffic

In Illumio Console, analyze the discovered traffic flows:

  • Use Traffic Map to validate expected traffic flows between workloads (for example, web-app and app-database).

  • Use Traffic Table to review the top peers and ports, and any flagged unexpected connections.

Tip

Read the Troubleshooting guide for help resolving issues.