Bulk Traffic Loader
This Public Experimental API allows you to upload a snapshot of static traffic connections from a non-Illumio environment to the PCE so you can visualize what your network traffic data looks like in the PCE Illumination map without having to install VENs on your workloads.
Note
Bulk operations are rate-limited to 1,000 items per operation.
API Methods
Functionality | HTTP | URI |
---|---|---|
Upload a collection of traffic data to the PCE |
|
|
Workflow to Upload Bulk Traffic
To upload bulk traffic data to the PCE:
Obtain network traffic information from your environment (using
netstat
or another network utility). For each traffic connection, obtain the source IP address, destination IP address, destination port, and protocol.Create unmanaged workloads in the PCE.
In the PCE, map both the source IP address and destination IP address for each traffic flow to the unmanaged workloads. An unmanaged workload is required for each source and destination IP address that represents traffic in the Illumination map. This must be done before you load the traffic data.
Use the Bulk Traffic Loader API to upload the CSV-formatted traffic data into the PCE.
Create Collection of Unmanaged Workloads
Using this method, you can create a collection of unmanaged workloads in your organization using a single API call, rather than having to individually create individual workloads one at a time.
When you create a collection of workloads using this method, these workloads are recorded in the PCE as “unmanaged,” which means that no VEN has been installed on the workloads and no policy can be applied to them.
Create an unmanaged workload for each source IP address and destination IP address before you use the Bulk Traffic Loader API.
URI for Creating a Collection of Unmanaged Workloads
PUT [api_version][org_href]/workloads/bulk_create
Request Body
When you create a collection of workloads, you pass a JSON request body that defines the workload details.
Although this example illustrates the request body for a single workload, you can add as many workloads as you want using the bulk_create
method.
Request Properties
JSON Payload
{ "name": "workload 0", "description": "workload desc 0", "service_principal_name": "spn 0", "hostname": "workload-0.example.com", "public_ip": "24.1.1.1", "ephemeral": false, "external_data_set": "cmdb", "external_data_reference": "0", "interfaces": [ { "name": "eth0", "link_state": "up", "address": "10.0.0.2", "cidr_block": 24, "ip_version": 4, "default_gateway_address": "10.0.0.1", "friendly_name": "wan" } ], "labels": [ {"href": "/orgs/2/labels/1"}, {"href": "/orgs/2/labels/9"} ], "service_provider": "service provider", "data_center": "central data center", "os_id": "os id 0", "os_detail": "os detail 0", "online": true, "agent": { "config": { "enforcement_mode": "full", "visibility_level": "flow_summary" } } }
Curl Command to Create Unmanaged Workload Collection
This curl example illustrates how to create two workloads using the Workload Bulk Create API. This example assumes you have already created API keys for PCE authentication.
curl -i -X PUT https://pce.mycompany.com:8443/api/v2/orgs/2/workloads/bulk_create -H "Content-Type:application/json" -u $KEY:$TOKEN -d '[{"name":"web_app1","description": "workload desc 0","service_principal_name":"spn 0","hostname":"workload-0.example.com", "public_ip":"24.1.0.1","ephemeral":false,"external_data_set":"cmdb","external_data_reference": "0","interfaces":[{"name":"eth0","link_state":"up","address":"10.0.0.2","cidr_block":24, "ip_version":4,"default_gateway_address":"10.0.0.1","friendly_name":"wan"}],"labels":[{"href":"/orgs/2/labels/1"},{"href":"/orgs/2/labels/9"}],"service_provider":"service provider","data_center":"central data center","os_id":"os id 0","os_detail": "os detail 0","online":true,"agent":{"config":{"enforcementmode":"full", "visibility_level":"flow_summary"}}}]'
Upload Bulk Traffic Flows
Note
Illumio recommends limiting bulk traffic operations to a maximum of 1,000 traffic flows per operation.
This method uses a POST command and requires a payload of multiple network traffic connections (source IP address, destination IP address, destination port, and protocol for each flow) passed as a single string of data in the CSV format.
URI for Uploading Bulk Traffic Flows
POST [api_version][org_href]/agents/bulk_traffic_flows
Header
When making this call, specify in the header that the data type is CSV and provide the CSV file format version. For example:
X-Bulk-Traffic-Load-CSV-Version:1
CSV Input Format
The traffic flow data that this method sent to the PCE in the CSV format must contain the following information:
Column | Type | Comment |
---|---|---|
| String (IP address) | Must be a correctly-formatted IPv4 or IPv6 address. This IP address must map to an unmanaged workload in the PCE before you load the traffic data to the PCE. |
| String (IP address) | Must be a correctly-formatted IPv4 or IPv6 address. This IP address must map to an unmanaged workload in the PCE before you load the traffic data to the PCE. |
| Integer | Destination port (port exposed by the destination IP address). |
| Integer | Protocol integer that must be converted to its Assigned Internet Protocol Numbers (for example., “tcp,” must be converted it to its IANA number 6) before you make the API call. |
Note
The network protocol for each connection must be translated to its assigned internet protocol number before you use this API to send the data. These protocol numbers can be found on the Internet Assigned Numbers Authority (IANA) website.
CSV Payload
For each network traffic connection, provide the source IP address, destination IP address, destination port, and network protocol (translated to its assigned internet protocol number). Each new connection entry must be specified by an \n
to indicate a line break in the CSV file.
Example of a flows.csv
file:
"192.3.106.142,192.3.66.238,10123,6\n192.3.106.142,192.5.179.150,5723,6\n192.3.199.114,192.4.114.14,49241,6 \ 192.4.113.239,192.4.113.240,1573,6\n192.4.205.21,192.1.189.138,13724,6\n192.4.205.21,192.1.189.138,1556,6 \ 192.1.189.137,192.1.189.136,1563,6\n192.1.189.137,192.1.189.136,1563,6\n192.1.189.137,192.1.189.135,6200,6 \ 192.1.189.137,192.1.189.136,1575,6\n"
Curl Commands to Upload Bulk Traffic Flows from CSV
If you have a pre-formatted CSV file, you can use the following curl command to upload the traffic flow data (where @/path/flows.csv
represents the network path of the .csv
file):
curl -i -X POST https://pce.illum.io:8443/api/v2/orgs/2/agents/bulk_traffic_flows -H "X-Bulk-Traffic-Load-CSV-Version: 1" -u $KEY:$TOKEN --data-binary "@/{path}/flows.csv"
This curl example shows two traffic connections being uploaded to the PCE:
curl -i -X POST https://mypce.mycompany.com:8443/api/v2/orgs/2/agents/bulk_traffic_flows -u $KEY:$TOKEN -H "X-Bulk-Traffic-Load-CSV-Version: 1" --data "10.0.0.1,10.0.0.2,23,6\n10.0.0.4,10.0.0.5,45,6"
This curl example shows a CSV file with traffic connections uploaded to the PCE:
curl -i -X POST https://devtest167.ilabs.io:8443/api/v2/orgs/1/agents/bulk_traffic_flows -T "bulk_upload.csv" -H "X-Bulk-Traffic-Load-CSV-Version: 1" -H "Content-Type: text/csv" -u $KEY:$TOKEN
Response Body
After successfully uploading traffic data with this API, an HTTP Status code of “201 Created” is returned along with a JSON object in the response body that contains a summary of the operation:
num_flows_received
: Total number of flows sent to the PCEnum_flows_failed
: Number of flows that failed to be uploadedfailed_flows
: Array of CSV lines that failed to parse or match a Workload
Example:
{ "num_flows_received": 10, "num_flows_failed": 0 "failed_flows": [] }