Setting up Illumio VEN on NVIDIA DPU
Follow this guide to set up and run an Illumio VEN on NVIDIA BlueField-3 DPU.
Prerequisites
NVIDIA prerequisites
NVIDIA BlueField DPU Image: v4.11.0 or later
NVIDIA BlueField DPU Firmware: v24.35.0388 or later
NVIDIA BlueField DPU Configuration: Follow NVIDIA docs to configure the DPU in
DPU Mode(also known as Embedded CPU Function Ownership (ECPF) mode).DOCA Installation: Follow NVIDIA docs to install DOCA for both host and DPU.
Illumio prerequisites
Host/VM OS: Linux
PCE: use the PCE provided by Illumio
Illumio VEN: use the VEN package provided by Illumio (
illumio-ven-25.2.40-4000.u16.arm64.deb)
Set up host
Set up your host for NVIDIA DOCA Argus.
Enable IOMMU passthrough
On your host, edit the GRUB configuration file to enable IOMMU passthrough.
Open the GRUB configuration file:
sudo vim /etc/default/grub
Add the following to
GRUB_CMDLINE_LINUX_DEFAULT:iommu=pt <intel/amd>_iommu=on
Save the file and apply the changes.
For example, on Ubuntu run
sudo update-grub.Reboot your host.
Prepare target system
Tip
Repeat this section once for each distinct target system environment whose memory the DPU will scan on the host system. A distinct environment is defined by a unique operating system and OS version (for example, Ubuntu 20.04 vs. Ubuntu 22.04, or different guest VM operating systems). You only need to repeat this preparation when the OS or OS version differs; otherwise, multiple systems running the same OS version can reuse the same prepared artifacts.
Prepare the target system configuration separately for each unique operating system and OS version the DPU will scan on the host system (for example, each distinct VM OS version running on the host).
Once the memory regions and OS symbols mappings are generated they can be reused for any target system of that OS version. If more than one type of system (VM/host) is being scanned, these regions and mappings may be added to the config folder of subdirectories for each supported system.
Once the memory region definitions and OS symbol mappings are generated, they can be reused for any target system running the same OS and OS version. If the DPU scans more than one type of target system (for example, host and VM), store the generated artifacts in separate subdirectories under the configuration folder, organized by OS and OS version.
Download OS debug symbols.
For Ubuntu:
sudo tee /etc/apt/sources.list.d/ddebs.list << EOF deb http://ddebs.ubuntu.com/ $(lsb_release -cs) main restricted universe multiverse deb http://ddebs.ubuntu.com/ $(lsb_release -cs)-updates main restricted universe multiverse deb http://ddebs.ubuntu.com/ $(lsb_release -cs)-proposed main restricted universe multiverse EOF sudo apt install ubuntu-dbgsym-keyring sudo apt-get update sudo apt-get install linux-image-$(uname -r)-dbgsym
Build
dwarf2json:Clone the dwarf2json repo.
Navigate to the repo and build:
cd dwarf2json && go build
Create a directory to store target symbols:
mkdir target_symbols
Create JSON files and copy them to
target_symbols:cd /opt/mellanox/doca/tools/ pip3 install psutil pdbparse sudo python3 /opt/mellanox/doca/tools/doca_apsh_config.py --files memregions symbols --os <windows/linux> --path <path to dwarf2json> cp /opt/mellanox/doca/tools/*.json ~illumio/target_symbols
Copy all files from
target_symbolsto the DPU viascpto a destination directory of your choice (for example,/etc/doca_argus_config/). The copied files should includeProfile_*.jsonandmem_regions.json.You will reference the destination directory (for example,
/etc/doca_argus_config) during DPU setup.
Configure NVIDIA DOCA Virtual Functions
Follow the NVIDIA guide to configure NVIDIA DOCA Virtual Functions (VFs on DPU, VF representors on host).
Tip
Refer to NVIDIA docs for help configuring SR-IOV virtual functions on the host.
Set up DPU
After preparing a target system, install the VEN on the DPU and run the DOCA Argus container.
Configure BlueField firmware
On the BlueField system, configure the PF base register and NVMe emulation settings:
mlxconfig -d /dev/mst/mt41686_pciconf0 s PF_BAR2_SIZE=2 PF_BAR2_ENABLE=1 NVME_EMULATION_ENABLE=1
To apply the settings, follow the NVIDIA guide to perform a BlueField system reboot.
After rebooting, you can run a command toverify whether the settings were successfully applied:
sudo mlxconfig -d /dev/mst/mt41686_pciconf0 q | grep -E "NVME|BAR"
The output should include the following configurations:
PF_BAR2_SIZE 2 PF_BAR2_ENABLE True(1) NVME_EMULATION_ENABLE True(1)
Configure huge pages for the DPU:
echo '1024' | sudo tee -a /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages sudo mkdir /mnt/huge sudo mount -t hugetlbfs -o pagesize=2M nodev /mnt/huge
Configure DOCA Argus
Tip
Refer to DOCA Argus Service Guide.
Modify
doca_argus.yaml:Under
volumes, add a volume namedconf-argus-configwith the path on the DPU at which thedoca_apsh_config.pyoutput files were saved to in step 2(d) of the Host Setup:volumes: - name: logs-argus-service hostPath: path: /var/log/doca_argus type: DirectoryOrCreate - name: logs-argus-activity-report hostPath: path: /var/log/doca_argus_activity_report type: DirectoryOrCreate - name: conf-argus-config #<--------------Add this volume hostPath: path: /etc/doca_argus_config #<-----------Path from step 2(d) type: DirectoryOrCreateUnder
volumeMounts, mount a volume namedconf-argus-configto a path of your choice:volumeMounts: - name: logs-argus-service mountPath: /var/log/doca_argus - name: logs-argus-activity-report mountPath: /var/log/doca_argus_activity_report - name: conf-argus-config mountPath: /etc/doca_argus_config #<------------ Choose a pathUnder
apsh_config, setos_symbol_pathandmen_regions_pathto be the same path you chose previously forconf-argus-config:apsh_config: # When true, scans for additional systems beyond those specified in config. Always enabled if systems section is empty auto_scan: true # Default values applied to all systems (including auto-scanned systems) unless overridden in the system section default: # OS symbol manifest path - can be single json, folder of jsons or auto. 'auto' means automatic detection os_symbol_path: /etc/doca_argus_config/ # Memory regions path. 'auto' means automatic detection mem_regions_path: /etc/doca_argus_config/ #<------------------- Same path as conf-argus-config
Copy
doca_argus.yamlto the path required to start the DOCA Argus container:cp doca_argus.yaml /etc/kubelet.d/
Configure OVS
Configure an OVS (Open vSwitch) bridge and attach an sFlow config to it to provide visibility to traffic between host and DPU.
Add representor devices to an OVS bridge.
Choose an OVS bridge name, for example
ovsbr2.Choose a sampling rate, for example
sampling=512.Choose your representor device(s).
An example representor device is
pf0hpf. If you have representors configured for host-side networking and you want to discover their corresponding workloads, you may configure those as well. Note: this configuration is functionally creating a bridge between representor devices.
ovs-vsctl add-br ovsbr2 ovs-vsctl add-port ovsbr2 <representor-device> ovs-vsctl -- --id=@sflow create sflow agent=oob_net0 target=\"127.0.0.1:6343\" sampling=512 polling=10 -- set bridge ovsbr2 sflow=@sflow
Install the VEN
Download and install the VEN:
apt install ./illumio-ven-25.2.40-4000.u16.arm64.deb
Set up the PCE
To enable workload discovery reporting via DOCA Argus, configure the Illumio PCE to use C-VEN (Containerized VEN) APIs.
Pair the VEN with the PCE.
On the PCE, create a container cluster node object.
Save the cluster UUID, Org ID, and Token (in
<org-id>_Tokenformat) for a subsequent step.Get the machine ID of the DPU where the VEN is deployed:
cat /etc/machine-id
Save the machine ID for a subsequent step.
Send a request to the PCE.
Use these options in the request:
cluster_uuid: use your saved cluster UUIDorg_id: use your saved cluster Org IDcluster_token_value: modify your saved cluster token by removing its org prefix.For example, given a cluster token of
2_123456789, remove the org prefix2_and use123456789for thecluster_token_value.
curl https://<your pce address>:8443/api/v2/orgs/<org_id>/container_clusters/<cluster_uuid>/put_from_cluster -v -u kubelink_<cluster_uuid>:<cluster_token_value> -X PUT -H "Content-Type: application/json" -d '{"nodes": [{"machine_id": "<your machine id>", "pod_subnet": "172.17.0.0/16"}]}'If you receive a 429 response code, you may need to increase the PCE object limits on the number of interfaces per agent.
Log in to the PCE and configure its interfaces per agent:
/opt/illumio-pce/illumio-pce-ctl obj-limits set --profile Default --key interfaces_per_agent --hard_limit 64
Create a Container Workload Profile on the PCE for the newly created container cluster.
Validate setup
Use Illumio Console to test and validate that your VEN on DPU is activated, workloads are discovered, and traffic is captured and analyzed.
Validate VEN activation
In Illumio Console, validate that VEN activation was successful:
VEN status is Active/Connected
The DPU host appears with the correct hostname, IP addresses, labels, and interfaces.
Validate workload discovery
In Illumio Console, validate that workloads have been discovered:
VMs on the DPU host are inventoried and labeled.
Downstream workloads via DPU-routed traffic are visible and discoverable.
Analyze discovered traffic
In Illumio Console, analyze the discovered traffic flows:
Use Traffic Map to validate expected traffic flows between workloads (for example, web-app and app-database).
Use Traffic Table to review the top peers and ports, and any flagged unexpected connections.
Tip
Read the Troubleshooting guide for help resolving issues.