Skip to main content

REST APIs for 22.2

Bulk Traffic Loader

This Public Experimental API allows you to upload a snapshot of static traffic connections from a non-Illumio environment to the PCE so you can visualize what your network traffic data looks like in the PCE Illumination map without having to install VENs on your workloads.

Note

Bulk operations are rate-limited to 1,000 items per operation.

API Methods

Functionality

HTTP

URI

Upload a collection of traffic data to the PCE

POST

[api_version][org_href]/agents/bulk_traffic_flows

Workflow to Upload Bulk Traffic

To upload bulk traffic data to the PCE:

  1. Obtain network traffic information from your environment (using netstat or another network utility). For each traffic connection, obtain the source IP address, destination IP address, destination port, and protocol.

  2. Create unmanaged workloads in the PCE.

  3. In the PCE, map both the source IP address and destination IP address for each traffic flow to the unmanaged workloads. An unmanaged workload is required for each source and destination IP address that represents traffic in the Illumination map. This must be done before you load the traffic data.

  4. Use the Bulk Traffic Loader API to upload the CSV-formatted traffic data into the PCE.

Create Collection of Unmanaged Workloads

Using this method, you can create a collection of unmanaged workloads in your organization using a single API call, rather than having to individually create individual workloads one at a time.

When you create a collection of workloads using this method, these workloads are recorded in the PCE as “unmanaged,” which means that no VEN has been installed on the workloads and no policy can be applied to them.

Create an unmanaged workload for each source IP address and destination IP address before you use the Bulk Traffic Loader API.

URI for Creating a Collection of Unmanaged Workloads

PUT [api_version][org_href]/workloads/bulk_create

Request Body

When you create a collection of workloads, you pass a JSON request body that defines the workload details.

Although this example illustrates the request body for a single workload, you can add as many workloads as you want using the bulk_create method.

Request Properties

JSON Payload

{
    "name": "workload 0",
    "description": "workload desc 0",
    "service_principal_name": "spn 0",
    "hostname": "workload-0.example.com",
    "public_ip": "24.1.1.1",
    "ephemeral": false,
    "external_data_set": "cmdb",
    "external_data_reference": "0",
    "interfaces": [
      {
        "name": "eth0",
        "link_state": "up",
        "address": "10.0.0.2",
        "cidr_block": 24,
        "ip_version": 4,
        "default_gateway_address": "10.0.0.1",
        "friendly_name": "wan"
      }
    ],
    "labels": [
      {"href": "/orgs/2/labels/1"},
      {"href": "/orgs/2/labels/9"}
    ],
    "service_provider": "service provider",
    "data_center": "central data center",
    "os_id": "os id 0",
    "os_detail": "os detail 0",
    "online": true,
    "agent": {
      "config": {
        "enforcement_mode": "full",
        "visibility_level": "flow_summary"
      }
    }
 }

Curl Command to Create Unmanaged Workload Collection

This curl example illustrates how to create two workloads using the Workload Bulk Create API. This example assumes you have already created API keys for PCE authentication.

curl -i -X PUT https://pce.mycompany.com:8443/api/v2/orgs/2/workloads/bulk_create -H "Content-Type:application/json" -u $KEY:$TOKEN -d '[{"name":"web_app1","description": "workload desc 0","service_principal_name":"spn 0","hostname":"workload-0.example.com", "public_ip":"24.1.0.1","ephemeral":false,"external_data_set":"cmdb","external_data_reference": "0","interfaces":[{"name":"eth0","link_state":"up","address":"10.0.0.2","cidr_block":24, "ip_version":4,"default_gateway_address":"10.0.0.1","friendly_name":"wan"}],"labels":[{"href":"/orgs/2/labels/1"},{"href":"/orgs/2/labels/9"}],"service_provider":"service provider","data_center":"central data center","os_id":"os id 0","os_detail": "os detail 0","online":true,"agent":{"config":{"enforcementmode":"full", "visibility_level":"flow_summary"}}}]' 
Upload Bulk Traffic Flows

Note

Illumio recommends limiting bulk traffic operations to a maximum of 1,000 traffic flows per operation.

This method uses a POST command and requires a payload of multiple network traffic connections (source IP address, destination IP address, destination port, and protocol for each flow) passed as a single string of data in the CSV format.

URI for Uploading Bulk Traffic Flows

POST [api_version][org_href]/agents/bulk_traffic_flows

Header

When making this call, specify in the header that the data type is CSV and provide the CSV file format version. For example:

X-Bulk-Traffic-Load-CSV-Version:1
CSV Input Format

The traffic flow data that this method sent to the PCE in the CSV format must contain the following information:

Column

Type

Comment

src

String (IP address)

Must be a correctly-formatted IPv4 or IPv6 address. This IP address must map to an unmanaged workload in the PCE before you load the traffic data to the PCE.

dst

String (IP address)

Must be a correctly-formatted IPv4 or IPv6 address. This IP address must map to an unmanaged workload in the PCE before you load the traffic data to the PCE.

destination port

Integer

Destination port (port exposed by the destination IP address).

protocol

Integer

Protocol integer that must be converted to its Assigned Internet Protocol Numbers (for example., “tcp,” must be converted it to its IANA number 6) before you make the API call.

Note

The network protocol for each connection must be translated to its assigned internet protocol number before you use this API to send the data. These protocol numbers can be found on the Internet Assigned Numbers Authority (IANA) website.

CSV Payload

For each network traffic connection, provide the source IP address, destination IP address, destination port, and network protocol (translated to its assigned internet protocol number). Each new connection entry must be specified by an \n to indicate a line break in the CSV file.

Example of a flows.csv file:

"192.3.106.142,192.3.66.238,10123,6\n192.3.106.142,192.5.179.150,5723,6\n192.3.199.114,192.4.114.14,49241,6 \
192.4.113.239,192.4.113.240,1573,6\n192.4.205.21,192.1.189.138,13724,6\n192.4.205.21,192.1.189.138,1556,6 \
192.1.189.137,192.1.189.136,1563,6\n192.1.189.137,192.1.189.136,1563,6\n192.1.189.137,192.1.189.135,6200,6 \
192.1.189.137,192.1.189.136,1575,6\n"

Curl Commands to Upload Bulk Traffic Flows from CSV

If you have a pre-formatted CSV file, you can use the following curl command to upload the traffic flow data (where @/path/flows.csv represents the network path of the .csv file):

curl -i -X POST https://pce.illum.io:8443/api/v2/orgs/2/agents/bulk_traffic_flows -H "X-Bulk-Traffic-Load-CSV-Version: 1" -u $KEY:$TOKEN --data-binary "@/{path}/flows.csv" 

This curl example shows two traffic connections being uploaded to the PCE:

curl -i -X POST https://mypce.mycompany.com:8443/api/v2/orgs/2/agents/bulk_traffic_flows -u $KEY:$TOKEN -H "X-Bulk-Traffic-Load-CSV-Version: 1" --data "10.0.0.1,10.0.0.2,23,6\n10.0.0.4,10.0.0.5,45,6"

This curl example shows a CSV file with traffic connections uploaded to the PCE:

curl -i -X POST https://devtest167.ilabs.io:8443/api/v2/orgs/1/agents/bulk_traffic_flows -T "bulk_upload.csv" -H "X-Bulk-Traffic-Load-CSV-Version: 1" -H "Content-Type: text/csv" -u $KEY:$TOKEN  

Response Body

After successfully uploading traffic data with this API, an HTTP Status code of “201 Created” is returned along with a JSON object in the response body that contains a summary of the operation:

  • num_flows_received: Total number of flows sent to the PCE

  • num_flows_failed: Number of flows that failed to be uploaded

  • failed_flows: Array of CSV lines that failed to parse or match a Workload

Example:

{ 
    "num_flows_received": 10,
    "num_flows_failed": 0 
    "failed_flows": []
}