Skip to main content

REST APIs for 23.5

Traffic Flow Summaries

This section describes traffic flow summaries.

After you install a VEN on a workload and pair the VEN with the PCE, the VEN monitors each workload's traffic flows and sends the traffic flow summaries to the PCE.

Traffic summaries can be exported to syslog or Fluentd. If traffic data is configured for export, the PCE processes the received traffic flow summaries from each VEN and immediately sends them to syslog or Fluentd.

Traffic Flow Types and Properties

The Illumio Core logs traffic flows based on the Visibility setting. Events have attributes that can be Allowed, Blocked, or Potentially Blocked and might not appear in the traffic flow summary.

Visibility Settings

The table below indicates whether or not a traffic summary is logged as Allowed, Potentially Blocked, or Blocked depending on a workload's policy state.

Note

Traffic from workloads in the “Idle” policy state is not exported to syslog from the PCE.

Visibility

Logged-in Traffic Flow Summary

Off

VEN does not log traffic connection information

Blocked - Low Detail

VEN logs connection information for blocked and potentially blocked traffic only

Blocked + Allowed - High Detail

VEN logs connection information for allowed, blocked, and potentially blocked traffic

Enhanced Data Collection

VEN logs byte counts in addition to connection details for allowed, blocked, and potentially blocked traffic

Event Types

In a traffic flow summary, the event type is designated by Policy Decision (pd).

Note

An asterisk ( * ) indicates the attribute might not appear in the summary.

Event Attributes

Allowed (pd=0)

Potentially Blocked (pd=1)

Blocked (pd=2)

Unknown (pd=3)

version

✓

✓

✓

✓

count

✓

✓

✓

✓

interval_sec

✓

✓

✓

✓

timestamp

✓

✓

✓

✓

dir

✓

✓

✓

✓

src_ip

✓

✓

✓

✓

dst_ip

✓

✓

✓

✓

proto

✓

✓

✓

✓

dst_prt

✓

✓

✓

✓

state

✓

✓

✓

✓

pd

✓

✓

✓

✓

code*

✓

✓

✓

✓

type*

✓

✓

✓

✓

dst_vulns*

✓

✓

✓

✓

fqdn*

✓

✓

✓

✓

un*

✓

✓

X

✓

pn*

✓

✓

X

✓

sn*

✓

✓

X

✓

src_labels*

✓

✓

✓

✓

dst_labels*

✓

✓

✓

✓

src_hostname*

✓

✓

✓

✓

dst_hostname*

✓

✓

✓

✓

src_href*

✓

✓

✓

✓

dst_href*

✓

✓

✓

✓

Show Amount of Data Transfer

The JSON, CEF, and LEEF for the accurate byte count work events are related to the 'Show Amount of Data Transfer' preview feature available with the Illumio Core 20.2.0 release.

The PCE now reports amount of data transferred in to and out of workloads and applications in a datacenter. The number of bytes sent by and received by the source of an application are provided separately. You can see these values in traffic flow summaries streamed out of the PCE. You can enable this capability on a per-workload basis in the Workload page. You can also enable it in the pairing profile so that workloads are directly paired into this mode.

The direction reported in flow summary is from the viewpoint of the source of the flow:

Destination Total Bytes Out: Number of bytes transferred out of source:

      dst_tbo

Destination Total Bytes In: Number of bytes transferred in to source.

      dst_tbi

To activate the 'Show Amount of Data Transfer' capability on the PCE, contact your Illumio representative.

LEEF Mapping

  • LEEF field X contains JSON field Y

  • srcBytes contains dst_tbo

  • dstBytes contains dst_tbi

  • dbi contains dst_dbi

  • dbo contains dst_dbo

CEF Mapping

  • CEF field cn2 is dst_dbi with cn2Label is “dbi”

  • CEF field cn3 is dst_dbo with cn3Label is “dbo”

  • CEF field “in” is dst_tbi

  • CEF field “out” is dst_tbo

Manage Traffic Flows Using REST API

You can use the following properties to manage traffic flows using the REST API.

Note

You should ignore and not use any extra properties that are not described in this document, such as tbi, tbo, dbi, and dbo.

Property

Description

Type

Required

Possible Values

version

The version of the flow summary schema.

Integer

Yes

4

timestamp

Indicates the time (RFC3339) when the first flow in the summary was created, represented in UTC.

Format: yyyy-MM-dd'T'HH:mm:ss.SSSSSSZ

String

Yes

interval_sec

Sample duration for the flows in the summary. Default is approximately 600 seconds (10 minutes), depending on the VEN's ability to report traffic and PCE's current load.

Integer

Yes

dir

Direction of the first packet: in or out (I, O).

String

Yes

I, O

src_ip

Source IP of the flows.

String

Yes

dst_ip

Destination IP of the flows.

String

Yes

proto

Protocol number (0-255).

Integer

Yes

Minimum=0

Maximum=255

type

The ICMP message type associated with the first flow in the summary. This value exists only if protocol is ICMP (1).

Note

This information is included in blocked flows for VEN versions lower than 19.1.0. It is included in all flows for VEN version 19.1.0 and later.

Example: 3 for “Destination Unreachable.”

Integer

No

Minimum=0

Maximum=255

code

The ICMP message code (subtype) associated with the first flow in the summary. This value exists only if protocol is ICMP (1).

Note

This information is included in blocked flows for VEN versions lower than 19.1.0. It is included in all flows for VEN version 19.1.0 and later.

Example: 1 for “Destination host unreachable.”

Integer

No

Minimum=0

Maximum=255

dst_port

Destination port.

This value exists only if protocol is not TCP (6) or UDP (17).

Integer

Yes

Minimum=0

Maximum=65535

pd

Policy decision value, which indicates if the flow was allowed, potentially blocked (but allowed), blocked, or unknown.

Possible values:

  • 0 – Allowed traffic

  • 1 – Allowed traffic but will be blocked after policy enforcement

  • 2 – Blocked traffic

  • 3 - Unknown

Note

Policy decision is “unknown” in the following cases:

  • Flows uploaded using existing bulk API (/orgs/<org_id>/agents/bulk_traffic_flows).

  • Flows uploaded using Network Flow Ingest Application (/orgs/<org_id>/traffic_data).

  • Traffic reported by idle VENs and specifically those that have been reported with “s” state (snapshot).

Integer

Yes

Minimum=0

Maximum=3

count

Count of the number of flows in the flow summary.

Integer

Yes

state

Session state for the traffic flows in the flow summaries.

Possible values:

  • Active (A): Connection was still open at the time the flow summary was logged. Applies to allowed and potentially blocked flows.

  • Closed (C): (Linux only) Connection closed at the time the flow summary was logged. Applies to allowed and potentially blocked flows.

  • Timed out (T): Connection timed out at the time the flow summary was logged. Applies to allowed and potentially blocked flows. Due to a limitation of WFP, a Windows VEN will report "T" even when the connection is closed at the time the flow summary was logged.

  • Snapshot (S): Snapshot of current connections to and from the VEN, which applies only to workloads whose policy state is set to Idle. Applies to allowed and potentially blocked flows.

  • New connection (N): Dropped TCP packet contains a SYN and is associated with a new connection. Applies to blocked TCP flows. The value is empty for blocked UDP flows.

String

No

A, C, T, S, N

pn

The program name is associated with the first flow of the summary. It is supported on inbound flows for Linux and Windows VEN and on outbound flows for only Windows VEN.

Note

This information might not be available on short-lived processes, which are Linux-specific.

Currently, flows are aggregated, so this value might represent only the first process detected across all aggregated flows.

If network communication is done by an OS component (or a driver), no process is associated with it.

String

No

un

The username is associated with the first flow of the summary. It is supported on inbound flows for Linux and Windows VEN and on outbound flows for only Linux VEN.

On Windows, it can include the username of the user account that initiated the connection.

Note

This information might not be available on short-lived processes.

String

No

sn

Service name associated with the first flow in the summary. It is supported only on inbound flows on Windows VEN.

String

No

src_hostname

Hostname of the source workload that reported the flow.

String

No

src_href

HREF of the source workload that reported the flow.

String

No

src_labels

Labels applied to the source workload.

Note

The src_hostname, src_href, and src_labels values are not be included in a traffic summary if the source of the flow is not an Illumio-labeled workload. For example, Internet traffic or a managed workload without any labels applied.

Object

No

dst_hostname

Hostname of the destination workload that reported the flow.

String

No

dst_href

HREF of the destination workload that reported the flow.

String

No

dst_labels

Labels applied to the destination workload.

Note

The dst_hostname, dst_href, and dst_labels values are not be included in a traffic summary if the destination of the flow is not an Illumio-labeled workload. For example, Internet traffic or a managed workload without any labels applied.

Object

No

dst_vulns

Information about the vulnerabilities on the destination of the traffic flow with the specific port and protocol.

Note

  • Vulnerabilities are defined by Common Vulnerabilities and Exposures (CVE), with identifiers and descriptive names from the U.S. Department of Homeland Security National Cybersecurity Center.

  • The vulnerability information is sent only when the Vulnerability Maps feature is turned on via a license and the information is imported into the PCE from a Vulnerability Scanner, such as Qualys.

Object

No

fqdn

Fully qualified domain name

String

No

The following table describes the sub-properties for the dst_vulns property:

Sub-property

Description

Type

Required

count

The total number of existing vulnerabilities on the destination port and protocol.

Integer

No

max_score

The maximum of all the scores for the vulnerabilities on the destination port and protocol.

Number

No

cve_ids

The list of CVE-IDs associated with the vulnerabilities that have the maximum score. Up to 100 displayed .

Array

No

Export Traffic Flow Summaries

Decide where to export the traffic flow summaries: syslog or Fluentd.

Caution

By default, from the 19.3.0 release on, the PCE generates all traffic flow summaries and sends them to syslog.

If you have not configured syslog, the syslog data by default is written to a local disk. For example, it is written to /var/log/messages.

Export to Syslog

To configure and export the traffic flow summaries to a remote syslog, follow these steps:

  1. From the PCE web console menu, choose Settings > Event Settings.

  2. Enable a remote syslog destination.

  3. Select specific traffic flow summaries to be sent to remote syslog.

    This filters the selected traffic flow summaries and send those to the remote syslog.

To prevent the syslog data from being written to a local disk based on your preference, deselect the Events checkboxes on the Settings > Event Settings > Local page in the PCE web console. For more information, see Events Settings.

Note

The generation of all traffic flow summaries is implemented to ensure that all of the traffic flow summaries are controlled from the PCE web console only.

This example shows the runtime_env.yml configuration to generate all types of flow summaries.

Export to Syslog

export_flow_summaries_to_syslog:
- accepted
- potentially_blocked
- blocked

This example shows the runtime_env.yml configuration if you do not want to generate any types of flow summaries.

Export to Syslog

export_flow_summaries_to_syslog:
- none

Note

Illumio does not currently support having a primary and secondary syslog configuration, with disaster recovery and failover.

You can configure it on a system syslog (local) and use the internal syslog configuration to send messages to local, which sends to system syslog.

Export to Fluentd

To generate and export the traffic flow summaries to Fluentd, follow these steps:

  1. Set the export_flow_summaries_to_fluentd parameter in runtime_env.yml.

  2. Set the external_fluentd_aggregator_servers parameter in runtime_env.yml.

This example shows the runtime_env.yml configuration to generate two types of flow summaries, out of the three possible types.

Export to Fluentd

external_fluentd_aggregator_servers:
- fluentd-server.domain.com:24224
export_flow_summaries_to_fluentd:
- accepted
- blocked
Flow Duration Attributes

The 20.2.0 VEN sends two new attributes to the syslog and fluentd output. The new attributes describe the flow duration and are appended to the flow data.

  • Delta flow duration in milliseconds (ddms): The duration of the aggregate within the current sampling interval. This field enables you to calculate the bandwidth between two applications in a given sampling interval. The formula is dbo (delta bytes out) / delta_duration_ms, or dbi / delta_duration_ms.

  • Total flow duration in milliseconds (tdms): The duration of the aggregate across all sampling intervals. This field enables you to calculate the average bandwidth of a connection between two applications. The formula is tbo (total bytes out) / total_duration_ms, or tbo / total_duration_ms. It also enables you to calculate the average volume of data in a connection between two applications. The formula is tbo (total bytes out) / count (number of flows in an aggregate), or tbi / count.

Traffic Flow Summary Examples

The following topic provides examples of traffic flow summaries in JSON, CEF, and LEEF, and messages that appear in syslog.

JSON
{
  "interval_sec": 600,
  "count": 1,
  "tbi": 73,
  "tbo": 0,
  "pn": "example-daemon",
  "un": "example",
  "src_ip": "xxx.xxx.xx.xxx",
  "dst_ip": "xxx.x.x.xxx",
  "timestamp": "2018-05-23T16:07:12-07:00",
  "dir": "I",
  "proto": 17,
  "dst_port": 5353,
  "state": "T",
  "src_labels": {
    "app": "AppLabel",
    "env": "Development",
    "loc": "Cloud",
    "role": "Web"
  },
  "src_hostname": "test-ubuntu-3",
  "src_href": "/orgs/1/workloads/xxxxxxxx-7741-4f71-899b-d6f495326b3f",
  "dst_labels": {
    "app": "AppLabel",
    "env": "Development",
    "loc": "AppLocation",
    "role": "Database"
  },
  "dst_hostname": "test-ubuntu-2",
  "dst_href": "/orgs/1/workloads/xxxxxxxx-012d-4651-b181-c6f2b269889e",
  "pd": 1,
  "dst_vulns": {
    "count": 8,
    "max_score": 8.5,
    "cve_ids": [
      "CVE-2016-2181",
      "CVE-2017-2241"
    ]
  },
  "fqdn" : "xxx.ubuntu.com",
  "version": 4
}
Syslog
2019-02-11T22:50:15.587390+00:00 level=info host=detest01 ip=100.1.0.1 
program=illumio_pce/collector| sec=925415.586 sev=INFO pid=9944
 tid=30003240 
rid=bb8ff798-1ef2-44b1-b74e-f13b89995520 {"interval_sec":1074,
"count":1,"tbi":3608,
"tbo":0,"pn":"company-daemon","un":"company","src_ip":"10.0.2.15",
"dst_ip":"211.0.0.232",
"class":"M","timestamp":"2019-02-11T14:48:09-08:00","dir":"I",
"proto":17,
"dst_port":5353,"state":"T","src_labels":{"app":"AppName",
"env":"Development","loc":"Cloud","role":"Web"},
"src_hostname":"dev-ubuntu-1",
"src_href":"/orgs/1/workloads/773f3e81-5779-4753-b879-35a1abe45838",
"dst_labels":{"app":"AppName","env":"Development","loc":"Cloud2",
"role":"Web"},
"dst_hostname":"dev-ubuntu-1","dst_href":"/orgs/1/workloads/
773f3e81-5779-4753-b879-35a1abe45838","pd":0,"dst_vulns":{"count":1,
"max_score":3.7,
"cve_ids":["CVE-2013-2566","CVE-2015-2808"]},"fqdn":"xxx.ubuntu.com",
"version":4}
Allowed Flow Summary (pd = 0)
2016-01-12T05:23:30+00:00 level=info host=myhost ip=127.0.0.1 program=illumio_pce/
collector| sec=576210.952 sev=INFO pid=25386 tid=16135120 rid=0 
{"interval_sec":1244,"count":3,"dbi":180,"dbo":180,"pn":"sshd","un":"root",
"src_ip":"10.6.0.129","dst_ip":"10.6.0.129","timestamp":"2017-08-16T13:23:57-07:00",
"dir":"I","proto":6,"dst_port":22,"state":"A","dst_labels":{"app":"test_app_1","env":
"test_env_1","loc":"test_place_1","role":"test_access_1"},"dst_hostname":"corp-vm-2",
"dst_href":"/orgs/1/workloads/5ddcc33b-b6a4-4a15-b600-64f433e4ab33","pd":0,
"version":4}
Potentially Blocked Flow Summary (pd = 1)
2016-01-12T05:29:21+00:00 level=info host=myhost ip=127.0.0.1 program=illumio_pce/
collector| sec=576561.327 sev=INFO pid=25386 tid=16135120 rid=0 sec=920149.541 
sev=INFO pid=1372 tid=30276700 rid=136019d0-f9d8-45f3-ac99-f43dd8015675 
{"interval_sec":600,"count":1,"tbi":229,"tbo":0,"src_ip":"172.16.40.5",
"dst_ip":"172.16.40.255","timestamp":"2017-08-16T14:45:58-07:00","dir":"I",
"proto":17,"dst_port":138,"state":"T","dst_labels":{"app":"test_app_1",
"env":"test_env_1","loc":"test_place_1","role":"test_access_1"},"dst_hostname":
"corp-vm-2","dst_href":"/orgs/1/workloads/5ddcc33b-b6a4-4a15-b600-64f433e4ab33",
"pd":1,"version":4}
Blocked Flow Summary (pd = 2)
2016-01-12T05:23:30+00:00 level=info host=myhost ip=127.0.0.1 program=illumio_pce/
collector| sec=576210.831 sev=INFO pid=25386 tid=16135120 rid=0 sec=915000.311 
sev=INFO pid=1372 tid=30302280 rid=90a01be5-a3c1-44f9-84fd-3c3a5eaec1f8 
{"interval_sec":589,"count":1,"src_ip":"10.6.1.89","dst_ip":"10.6.255.255",
"timestamp":"2017-08-16T13:22:09-07:00","dir":"I","proto":17,"dst_port":138,
"dst_labels":{"app":"test_app_1","env":"test_env_1","loc":"test_place_1",
"role":"test_access_1"},"dst_hostname":"corp-vm-1","dst_href":"/orgs/1/workloads/
a83ba658-576b-4946-800a-b39ba2a2e81a","pd":2,"version":4}
Unknown Flow Summary (pd = 3)
2019-06-14T05:33:45.442561+00:00 level=info host=devtest0 ip=127.0.0.1 
program=illumio_pce/collector| sec=490425.442 sev=INFO pid=12381 tid=32524120 
rid=6ef5a6ac-8a9c-4f46-9180-c0c91ef94759 {"dst_port":1022,"proto":6,"count":20,
"interval_sec":600,"timestamp":"2019-06-06T21:03:57Z","src_ip":"10.23.2.7",
"dst_ip":"10.0.2.15","dir":"O","state":"S","pd":3,"src_href":"/orgs/1/workloads/
a0d735ce-c55f-4a38-965f-bf6e98173598","dst_hostname":"workload1",
"dst_href":"/orgs/1/workloads/a20eb1b5-10a4-419e-b216-8b35c795a01e","src_labels":
{"app":"app","env":"Development","loc":"Amazon","role":"Load Balancer"}
,"version":4}
CEF
CEF:0|Illumio|PCE|2015.9.0|flow_potentially_blocked|Flow Potentially Blocked|3| 
act=potentially_blocked cat=flow_summary deviceDirection=0 dpt=137 src=someIPaddress 
dst=someIPaddress proto=udp cnt=1 in=1638 out=0 rt=Jun 14 2018 01:50:14 
cn1=120 cn1Label=interval_sec cs2=T cs2Label=state cs6=/orgs/1/workloads/
someID cs6Label=dst_href cs4={"app":"CRM","env":"Development","loc":"AppLocation",
"role":"Web"} cs4Label=dst_labels dhost=connectivity-check.someDomainName 
cs1={"count":1,"max_score":3.7,"cve_ids": ["CVE-2013-2566","CVE-2015-2808"]} 
cs1Label=dst_vulns dvchost=someDomainName
Unknown Flow Summary (pd = 3)
2019-06-14T21:02:55.146101+00:00 level=info host=devtest0 ip=127.0.0.1 
program=illumio_pce/collector| sec=546175.145 sev=INFO pid=15416 tid=40627440
 rid=f051856d-b9ee-4ac8-85ea-4cb857eefa82 CEF:0|Illumio|PCE|19.3.0|flow_unknown|
Flow Unknown|1|act=unknown cat=flow_summary deviceDirection=0 dpt=22 src=10.0.2.2
 dst=10.0.2.15 proto=tcp cnt=6 in=6 out=6 rt=Jun 14 2019 21:02:25 duser=root 
dproc=sshd cn1=31 cn1Label=interval_sec cs2=S cs2Label=state dhost=workload1 
cs6=/orgs/1/workloads/a20eb1b5-10a4-419e-b216-8b35c795a01e cs6Label=dst_href 
dvchost=devtest0.ilabs.io msg=
{"trafclass_code":"U"}
LEEF
LEEF:2.0|Illumio|PCE|2015.9.0|flow_blocked|cat=flow_summary 
devTime=2018-06-14T10:38:53-07:00 devTimeFormat=yyyy-MM-dd'T'HH:mm:ssX 
proto=udp sev=5 src=someIPaddress dst=someIPaddress dstPort=5353 count=15 
dir=I intervalSec=56728 dstHostname=someHostName dstHref=/orgs/1/workloads/
someID dstLabels={"app":"CRM","env":"Development","loc":"Cloud","role":"Web"} 
dstVulns={"count":2,"max_score":3.7} dstFqdn=someDomainName "cve_ids":
["CVE-2013-2566","CVE-2015-2808"]}
Unknown Flow Summary (pd = 3)
2019-06-14T19:25:53.524103+00:00 level=info host=devtest0 ip=127.0.0.1 
program=illumio_pce/collector| sec=540353.474 sev=INFO pid=9960 tid=36072680
 rid=49626dfa-d539-4cff-8999-1540df1a1f61 LEEF:2.0|Illumio|PCE|19.3.0|
flow_unknown|cat=flow_summary devTime=2019-06-06T21:03:57Z 
devTimeFormat=yyyy-MM-dd'T'HH:mm:ssX proto=tcp sev=1 src=10.23.2.7 
dst=10.0.2.15 dstPort=1022 count=20 dir=O intervalSec=600 state=S 
srcHref=/orgs/1/workloads/a0d735ce-c55f-4a38-965f-bf6e98173598 srcLabels=
{"app":"app","env":"Staging","loc":"Azure","role":"API"}
dstHostname=workload1 dstHref=/orgs/1/workloads/a20eb1b5-10a4-419e-b216-8b35c795a01e