Skip to main content

What's New and Release Notes for 26.x

Troubleshooting

Resolve common issues with Illumio VEN on NVIDIA BlueField-3 DPU.

No VMs shown in PCE

Problem: No VMs on your host system show up in the PCE.

Causes: You may have misconfigured DOCA Argus and/or representor devices.

Fixes: Try the following steps to diagnose and resolve the issue:

  • Verify the Virtual Function is correctly attached to the VM.

    • Use lspci:

      lspci -v | grep -i "virtual function"

      Example expected output:

      00:04.0 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function (rev 01)        Subsystem: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function

    • Use readlink:

      • Identify the name of the network interface which should represent the virtual function.

        readlink /sys/class/net/<network_device>/device/driver

        Example expected output:

        ../../../bus/pci/drivers/mlx5_core

      • Verify the mlx5_core driver is attached as in the output above.

  • Verify the network configuration on the DPU is set up such that the virtual function is reachable.

    • On the DPU, use ping to the virtual function's IP address on the VM:

      ping <virtual_function_ip_address>

      Verify the pings are successful.

  • Verify DOCA Argus is able to identify and read from the representor devices.

    • On the DPU, check the running DOCA Argus process:

      ps aux | grep doca_argus

    • If the output shows <defunct> processes, then the DOCA Argus' auto scan feature failed to read the representor device. Fix this by manually specifying the representor devices in doca_argus.yaml:

      • Under apsh_config, set auto_scan: false.

      • Identify all representor network interfaces on the system and generate a standardized name for each one that you'll add to doca_argus.yaml:

        • Follow NVIDIA docs to get representor_id and dma_device_name per system configuration.

        • For each representor ID of a representor attached to a VM, include the following fields under systems in doca_argus.yaml:

          <vm-name>:
  representor_id: <representor_id>
  mem_regions_path: /etc/doca_argus_config/
  os_symbol_path: /etc/doca_argus_config/
  dma_device_name: mlx5_1
  os_type: linux

          • Each <representor_id> must match a corresponding ID output by the get_representor_names.sh script.

          • Example: an excerpt of doca_argus.yaml with an updated systems field:

            systems:
  dpu-vm1:
    representor_id: IL0HFWRM7403146A00I1MLNXS0D0F1VF1
    mem_regions_path: /etc/doca_argus_config/
    os_symbol_path: /etc/doca_argus_config/
    dma_device_name: mlx5_1
    os_type: linux
  dpu-vm2:
    representor_id: IL0HFWRM7403146A00I1MLNXS0D0F1VF2
    mem_regions_path: /etc/doca_argus_config/
    os_symbol_path: /etc/doca_argus_config/
    dma_device_name: mlx5_1
    os_type: linux

  • Verify the appropriate host profile and memory regions JSON files have been provided in the DOCA Argus config directory.

    • Inspect /var/log/doca_argus_activity_report/<representor_id>.log.

      If the log contains Host Initialization Failed, Argus couldn't find the appropriate host profile.  To debug, reference the DOCA Argus Service Guide for more information.

No flow data captured

Problem: You don't observe flow data from the VEN on DPU.

Causes: sFlow and/or OVS bridge may not be configured or running as expected.

Fixes: Try the following steps to diagnose and resolve the issue:

  • Follow NVIDIA docs for general configuration checks of OVS inside BlueField.

  • Verify the DPU is running in Separated Host (DPU) mode:

    mlxconfig -d /dev/mst/mt41692_pciconf0 q | grep INTERNAL_CPU_MODEL

    Note

    Values for DPU running mode:

    • 0: Embedded (NIC Mode) default

    • 1: DPU Mode (Separated Host)

    • 2: Restricted Mode

    If the output is not DPU Mode (value: 1), run the following to set the running mode:

    mlxconfig -d /dev/mst/mt41692_pciconf0 s INTERNAL_CPU_MODEL=1

  • Verify the ovs-vsctl command is installed:

    which ovs-vsctl

    If not installed, install it:

    sudo apt-get install doca-openvswitch-switch doca-openvswitch-common

  • Verify OVS bridge creation.

    • Check that the OVS bridge is running:

      ovs-vsctl show

      In the output, verify your named bridge (for example, ovsbr2) is listed with your ports listed.

  • Verify port and representor additions.

    • List all ports on your bridge (named ovsbr2, for example) and show port details:

      ovs-vsctl list-ports ovsbr2
ovs-ofctl show ovsbr2

    • Verify the representor's status on the host is UP:

      ip link show <representor-device>

    • Verify traffic counters are incrementing:

       ip -s link show <representor-device>

  • Verify sFlow configuration.

    • Check for valid entries in the sFlow table of OVS:

      ovs-vsctl list sflow

      Expected output:

      - agent: "oob_net0"
- targets: ["127.0.0.1:6343"]
- sampling: 512
- polling: 10

    • Verify the sFlow collector is listening:

      netstat -ulnp | grep 6343

      or

      ss -ulnp | grep 6343

    • Capture and verify sflow packets are sent:

      tcpdump -i lo port 6343 -n

      Expect to capture UDP packets if sampling is working.

    • Check OVS logs for errors:

      tail -f /var/log/openvswitch/ovs-vswitchd.log
tail -f /var/log/openvswitch/ovsdb-server.log

Unable to configure SR-IOV Virtual Functions

Problem: Configuration of virtual functions fails.

Cause: SR-IOV may not be enabled.

Fix: Follow NVIDIA docs to verify that SR-IOV is enabled.

IOMMU and Passthrough issues

Problem: Your DPU cannot access host memory, or DOCA Argus fails to perform memory introspection.

Cause: The IOMMU may not be enabled on the DPU

Fix: Try the following steps to diagnose and resolve the issue:

  • Verify IOMMU is enabled in the kernel command line:

    cat /proc/cmdline | grep iommu

    Expected output should include iommu=pt and either intel_iommu=on or amd_iommu=on. If missing, follow the setup steps to Enable IOMMU Passthrough by editing /etc/default/grub .

  • Confirm the IOMMU groups are populated:

    ls /sys/kernel/iommu_groups/

    If the output is empty, the IOMMU isn't active despite being set. To resolve, check your BIOS/UEFI settings and verify that either VT-d (Intel) or AMD-Vi (AMD) are enabled.

DPU firmware configuration issues

Problem: The DOCA Argus container fails to start, you encounter NVMe emulation errors, or you observe BAR-related failures in logs.

Cause: Your DPU firmware may be misconfigured.

Fix: Try the following steps to diagnose and resolve the issue:

  • Verify DPU firmware settings have been applied:

    sudo mlxconfig -d /dev/mst/mt41686_pciconf0 q | grep -E "NVME|BAR"

    Expected output:

    PF_BAR2_SIZE            2
PF_BAR2_ENABLE          True(1)
NVME_EMULATION_ENABLE   True(1)

    If any output value is unexpected, run the following command to set firmware options:

    mlxconfig -d /dev/mst/mt41686_pciconf0 s PF_BAR2_SIZE=2 PF_BAR2_ENABLE=1 NVME_EMULATION_ENABLE=1

    After setting firmware options, follow NVIDIA docs to do a full system reboot.

    • If the MST device path (/dev/mst/mt41686_pciconf0) doesn't exist, run mst start and mst status to find the correct device path to your hardware.

Hugepage-related failures

Problem: DOCA Argus crashes or fails to allocate memory, with log messages about hugepage allocation failure.

Cause: Your DPU hugepage allocation may need to be increased.

Fix: Try the following steps to diagnose and resolve the issue:

  • Verify that your current hugepage allocation is at least 1024

    cat /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages

    If less than 1024, the system may not have had enough contiguous memory at allocation time. To fix, either retry the step to set hugepage allocation, or set it as a kernel parameter (hugepages=1024) in the boot process.

  • Verify the hugetlbfs mount:

    mount | grep huge

    The output should show hugetlbfs mounted at /mnt/huge. If that's missing, run the following to mount it:

    sudo mkdir -p /mnt/huge
sudo mount -t hugetlbfs -o pagesize=2M nodev /mnt/huge

    Note

    To make hugepage configuration persist across reboots, add it to /etc/fstab and a sysctl config or kernel boot parameter rather than relying on the echo command.To make hugepage configuration persist across reboots, add it to /etc/fstab and a sysctl config or kernel boot parameter rather than relying on the echo command.