Skip to main content

REST APIs for 23.5

Events Described

This section describes the concepts and types of PCE events.

Event Types, Syntax, and Record Format

When working with events, it is important to recognize their type, REST API schema, syntax, and record information.

Types of Events

The Illumio Core includes the following general categories of auditable events:

  • Organizational events: Organizational events are further grouped by their source:

    • API-related events: Events occurring from a use of the REST API, including the PCE web console

    • System-related events: Events caused by some system-related occurrence

  • Traffic events

Anonymized Database Dumps

To troubleshoot customer-reported issues, Illumio Customer Support sometimes requests that you supply an anonymized dump of the PCE database.

To safeguard your organization's privacy, the event information is not included in the anonymized database dump.

REST API Events Schema

The Events schema in JSON is downloadable from this documentation portal in the zipfile of the REST API schemas. From the documentation portal Home page, go to the Develop category > REST API Public Schemas (Archive File).

Event Syntax

The names of recorded auditable events in have the following general syntax:

resource.verb[.success_or_failure]

Where:

  • resource is a PCE and VEN object, such as PCE user or VEN agent component.

  • verb describes the action of the event on that resource.

  • In CEF and LEEF formats, the success or failure of the verb is included in the recorded event type. This indicator is not needed in the JSON format.

Events Record Information

The following information is included in a event record, which answers the who, what, where, how, and when:

Type of information

Description

Who

  • VEN identified by hostname and agent href, and after Release 22.3, VEN href

  • User identified by username and href

  • PCE system identified by “system”

What

The action that triggered the event, including the following data:

  • Resource type + operation + success or failure

  • Application Request ID

  • Status of successful events and failed events:

    • In case of failure, exception type and exception message.

    • All failures related to security, such as authentication and authorization.

    • Severity as INFO, WARNING, ERROR.

  • The pre-change and post-change values of the affected resources.

Where

The target resource of the action, composed of the following data:

  • Identifier of the target resource (primary field).

  • Friendly name for the target resource. For example:

    • workload/VEN: hostname

    • user.username

    • ruleset, label, service, etc: name, key/value

How

API endpoint, method, HTTP status code, and source IP address of the request.

When

Timestamp of the event's occurrence. This timestamp is not the time the event was recorded.

Event Record Structure

Regardless of export format (JSON, CEF, or LEEF), the records and fields for all events share a common structure. This common structure of composite events makes post-processing of event data easier.

Bulk change operations on many resources simultaneously are recorded as individual operations on the resource within a single composite event. Failed attempts to change a configuration, such as incorrect authentication, are also collected.

Common Fields

Field Name

Description

href

Unique event identifier; contains a UUID.

timestamp

Exact time that the event occurred in RFC 3339 format with fractional seconds.

pce_fqdn

The fully qualified domain name of the PCE; especially useful for Supercluster deployments or if there are multiple PCEs sending data to the SIEM server.

created_by

Identifies creator of the event; could be a user, the system, or a workload.

event_type

Name of the event; for more information, see the List of Event Types table.

status

“Success” or “failure;” if the status is null, the event is for information only and doesn't indicate success or failure.

severity

“Informational,” “warning,” or “error” indicating the severity of the event.

version

Schema version for events.

Events Displayed in PCE Web Console

The PCE web console provides an ongoing log of all Organization events that occur in the PCE. For example, Organization events capture actions such as users logging in and logging out, and failed login attempts; when a system object is created, modified, deleted, or provisioned; when a workload is paired or unpaired; and so on.

From the platform and API perspective, Organization events are referred to internally as auditable_events and are generated by the auditable_events_service.

You can use the filter at the top of the page to search for events by type of event, event severity level, and when the event occurred.

Cross-Site Request Forgery Protection

A cross-site request forgery (CSRF) is an attack that involves forcing a victim to send an HTTP request to a target destination without their knowledge or intent in order to perform an action as the victim. The underlying cause is an application functionality using predictable URL or form actions in a repeatable way. The nature of the attack is that CSRF exploits the trust that a website has for a user.

For more details on this attack, see the CSRF article on the Web Application Security Consortium website.

Illumio Core can notify you of this type of attack in the following ways:

  • The PCE web console logs the attack as an Organization Event called “CSRF token validation failure.”

  • The event is logged in the Illumio Core REST API as authz_csrf_validation_failure in the audit_log_events_get.schema.

  • The event authz_csrf_validation_failure appears in the PCE syslog output if you have deployed the PCE as a software.

Important

When you see this event occur, you should immediately investigate the issue because the request might not have originated from a valid user.

List of Event Types

The following table provides the types of JSON events generated and their description. For each of these events, the CEF/LEEF success or failure events generated are the event name followed by .success or .failure.

For example, the CEF/LEEF success event for agent.activate is agent.activate.success and the failure event is agent.activate.failure.

Each event can generate a variety of notification messages. See Notification Messages in Events.

JSON Event Type

Description

access_restriction.create

Access restriction created

access_restriction.delete

Access restriction deleted

access_restriction.update

Access restriction updated

agent.activate

Agent paired

agent.activate_clone

Agent clone activated

agent.clone_detected

Agent clone detected

agent.deactivate

Agent unpaired

agent.generate_maintenance_token

Generate maintenance token for any agent

agent.goodbye

Agent disconnected

agent.machine_identifier

Agent machine identifiers updated

agent.refresh_token

Agent refreshed token

agent.reguest_policy

Policy request sent

agent.request_upgrade

VEN upgrade request sent

agent.service_not_available

Agent reported a service not running

agent.suspend

Agent suspended

agent.tampering

Agent firewall tampered

agent.unsuspend

Agent unsuspended

agent.update

Agent properties updated.

agent.update_interactive_users

Agent interactive users updated

agent.update_iptables_href

Agent updated existing iptables href

agent.update_running_containers

Agent updated existing containers

agent.upload_existing_ip_table_rules

Agent existing IP tables uploaded

agent.upload_support_report

Agent support report uploaded

agent_support_report_request.create

Agent support report request created

agent_support_report_request.delete

Agent support report request deleted

agents.clear_conditions

Condition cleared from a list of VENs

agents.unpair

Multiple agents unpaired

api_key.create

API key created

api_key.delete

API key deleted

api_key.update

API key updated

auth_security_principal.create

RBAC auth security principal created

auth_security_principal.delete

RBAC auth security principal deleted

auth_security_principal.update

RBAC auth security principal updated

authentication_settings.update

Authentication settings updated

cluster.create

PCE cluster created

cluster.delete

PCE cluster deleted

cluster.update

PCE cluster updated

container_workload.update

Container workload updated

container_cluster.create

Container cluster created

container_cluster.delete

Container cluster deleted

container_cluster.update

Container cluster updated

container_cluster.update_label_map

Container cluster label mappings updated all at once

container_cluster.update_services

Container cluster services updated, created, or deleted by Kubelink

container_workload_profile.create

Container workload profile created

container_workload_profile.delete

Container workload profile deleted

container_workload_profile.update

Container workload profile updated

database.temp_table_autocleanup_started

DB temp table cleanup started

database.temp_table_autocleanup_completed

DB temp table cleanup completed

domain.create

Domain created

domain.delete

Domain deleted

domain.update

Domain updated

enforcement_boundary.create

Enforcement boundary created

enforcement_boundary.delete

Enforcement boundary deleted

enforcement_boundary.update

Enforcement boundary updated

event_settings.update

Event settings updated

firewall_settings.update

Global policy settings updated

group.create

Group created

group.update

Group updated

ip_list.create

IP list created

ip_list.delete

IP list deleted

ip_list.update

IP list updated

ip_lists.delete

IP lists deleted

ip_tables_rule.create

IP tables rules created

ip_tables_rule.delete

IP tables rules deleted

ip_tables_rule.update

IP tables rules updated

job.delete

Job deleted

label.create

Label created

label.delete

Label deleted

label.update

Label updated

label_group.create

Label group created

label_group.delete

Label group deleted

label_group.update

Label group updated

labels.delete

Labels deleted

ldap_config.create

LDAP configuration created

ldap_config.delete

LDAP configuration deleted

ldap_config.update

LDAP configuration updated

ldap_config.verify_connection

LDAP server connection verified

license.delete

License deleted

license.update

License updated

login_proxy_ldap_config.create

Interservice call to login service to create LDAP config

login_proxy_ldap_config.delete

Interservice call to login service to delete LDAP config

login_proxy_ldap_config.update

Interservice call to login service to update LDAP config

login_proxy_ldap_config.verify_connection

Interservice call to login service to verify connection to the LDAP server

login_proxy_msp_tenants.create

New MSP tenant created

login_proxy_msp_tenants.delete

MSP tenant deleted

login_proxy_msp_tenants.update

MSP tenant updated

login_proxy_orgs.create

New managed organization created

login_proxy_orgs.delete

Managed organization deleted

login_proxy_orgs.update

Managed organization updated

lost_agent.found

Lost agent found

network.create

Network created

network.delete

Network deleted

network.update

Network updated

network_device.ack_enforcement_instructions_applied

Enforcement instruction applied to a network device

network_device.assign_workload

Existing or new unmanaged workload assigned to a network device

network_device.create

Network device created

network_device.delete

Network device deleted

network_device.update

Network device updated

network_devices.ack_multi_enforcement_instructions_applied

Enforcement instructions applied to multiple network devices

network_endpoint.create

Network endpoint created

network_endpoint.delete

Network endpoint deleted

network_endpoint.update

Network endpoint updated

network_enforcement_node.activate

Network enforcement node activated

network_enforcement_node.clear_conditions

Network enforcement node conditions cleared

network_enforcement_node.deactivate

Network enforcement node deactivated

network_enforcement_node.degraded

Network enforcement node failed or primary lost connectivity to secondary

network_enforcement_node.missed_heartbeats

Network enforcement node did not heartbeat for more than 15 minutes

network_enforcement_node.missed_heartbeats_check

Network enforcement node missed heartbeats check

network_enforcement_node.network_devices_network_endpoints_workloads

Workload added to network endpoint

network_enforcement_node.policy_ack

Network enforcement node acknowledgment of policy

network_enforcement_node.request_policy

Network enforcement node policy requested

network_enforcement_node.update_status

Network enforcement node reports when switches are not reachable

network_enforcement_nodes.clear_conditions

A condition was cleared from a list of network enforcement nodes

nfc.activate

Network function controller created

nfc.delete

Network function controller deleted

nfc.update_discovered_virtual_servers

Network function controller virtual servers discovered

nfc.update_policy_status

Network function controller policy status

nfc.update_slb_state

Network function controller SLB state updated

org.create

Organization created

org.recalc_rules

Rules for organization recalculated

org.update

Organization information updated

pairing_profile.create

Pairing profile created

pairing_profile.create_pairing_key

Pairing profile pairing key created

pairing_profile.delete

Pairing profile deleted

pairing_profile.update

Pairing profile updated

pairing_profile.delete_all_pairing_keys

Pairing keys deleted from pairing profile

pairing_profiles.delete

Pairing profiles deleted

password_policy.create

Password policy created

password_policy.delete

Password policy deleted

password_policy.update

Password policy updated

permission.create

RBAC permission created

permission.delete

RBAC permission deleted

permission.update

RBAC permission updated

radius_config.create

Create domain RADIUS configuration

radius_config.delete

Delete domain RADIUS configuration

radius_config.update

Update domain RADIUS configuration

radius_config.verify_shared_secret

Verify RADIUS shared secret

request.authentication_failed

API request authentication failed

request.authorization_failed

API request authorization failed

request.internal_server_error

API request failed due to internal server error

request.service_unavailable

API request failed due to unavailable service

request.unknown_server_error

API request failed due to unknown server error

resource.create

Login resource created

resource.delete

Login resource deleted

resource.update

Login resource updated

rule_set.create

Rule set created

rule_set.delete

Rule set deleted

rule_set.update

Rule set updated

rule_sets.delete

Rule sets deleted

saml_acs.update

SAML assertion destination services updated

saml_config.create

SAML configuration created

saml_config.delete

SAML configuration deleted

saml_config.pce_signing_cert

Generate a new cert for signing SAML AuthN requests

saml_config.update

SAML configuration updated

saml_sp_config.create

SAML Service Provider created

saml_sp_config.delete

SAML Service Provider deleted

saml_sp_config.update

SAML Service Provider updated

sec_policy.create

Security policy created

sec_policy_pending.delete

Pending security policy deleted

sec_policy.restore

Security policy restored

sec_rule.create

Security policy rules created

sec_rule.delete

Security policy rules deleted

sec_rule.update

Security policy rules updated

secure_connect_gateway.create

SecureConnect gateway created

secure_connect_gateway.delete

SecureConnect gateway deleted

secure_connect_gateway.update

SecureConnect gateway updated

security_principal.create

RBAC security principal created

security_principal.delete

RBAC security principal bulk deleted

security_principal.update

RBAC security principal bulk updated

security_principals.bulk_create

RBAC security principals bulk created

service.create

Service created

service.delete

Service deleted

service.update

Service updated

service_account.create

Service account created

service_account.delete

Service account deleted

service_account.update

Service account updated

service_binding.create

Service binding created

service_binding.delete

Service binding created

service_bindings.delete

Service bindings deleted

service_bindings.delete

Service binding deleted

services.delete

Services deleted

settings.update

Explorer settings updated

slb.create

Server load balancer created

slb.delete

Server load balancer deleted

slb.update

Server load balancer updated

support_report.upload

Support report uploaded

syslog_destination.create

syslog remote destination created

syslog_destination.delete

syslog remote destination deleted

syslog_destination.update

syslog remote destination updated

system_task.agent_missed_heartbeats_check

Agent missed heartbeats

system_task.agent_missing_heartbeats_after_upgrade

VEN missing heartbeat after upgrade

system_task.agent_offline_check

Agents marked offline

system_task.agent_self_signed_certs_check

VEN self signed certificate housekeeping check

system_task.agent_settings_invalidation_error_state_check

VEN settings invalidation error state check

system_task.agent_uninstall_timeout

VEN uninstall timeout

system_task.clear_auth_recover_condition

Clear VEN authentication recovery condition

system_task.compute_policy_for_unmanaged_workloads

Compute policy for unmanaged workloads

system_task.delete_expired_service_account_api_keys

An expired service account api_key was successfully deleted

system_task.delete_old_cached_perspectives

Delete old cached perspectives

system_task.endpoint_offline_check

Endpoint marked offline

system_task.provision_container_cluster_services

Container cluster services provisioned

system_task.prune_old_log_events

Event pruning completed

system_task.remove_stale_zone_subsets

Stale zone subnets removed

system_task.set_server_sync_check

Set server synced

system_task.vacuum_deactivated_agent_and_deleted_workloads

Deactivated and deleted workloads have been vacuumed

traffic_collector_setting.create

Traffic collector setting created

traffic_collector_setting.delete

Traffic collector setting deleted

traffic_collector_setting.update

Traffic collector setting updated

trusted_proxy_ips.update

Trusted proxy IPs created or updated

user.accept_invitation

User invitation accepted

user.authenticate

User authenticated

user.create

User created

user.delete

User deleted

user.invite

User invited

user.login

User logged in

user.login_session_terminated

User login session terminated

user.logout

User logged

user.pce_session_terminated

User session terminated

user.reset_password

User password reset

user.sign_in

User session created

user.sign_out

User session terminated

user.update

User information updated

user.update_password

User password updated

user.use_expired_password

User entered expired password

user.verify_mfa

User verified MFA

users.auth_token

Auth token returned for user authentication on PCE

user_local_profile.create

User local profile created

user_local_profile.delete

User local profile deleted

user_local_profile.reinvite

User local profile reinvited

user_local_profile.update_password

User local password updated

ven_settings.update

VEN settings updated

ven_software.upgrade

VEN software release upgraded

ven_software_release.create

VEN software release created

ven_software_release.delete

VEN software release deleted

ven_software_release.deploy

VEN software release deployed

ven_software_release.update

VEN software release updated

ven_software_releases.set_default_version

Default VEN software version set

virtual_server.create

Virtual server created

virtual_server.delete

Virtual server created

virtual_server.update

Virtual server updated

virtual_service.create

Virtual service created

virtual_service.delete

Virtual service deleted

virtual_service.update

Virtual service updated

virtual_services.bulk_create

Virtual services created in bulk

virtual_services.bulk_update

Virtual services updated in bulk

vulnerability.create

Vulnerability record created

vulnerability.delete

Vulnerability record deleted

vulnerability.update

Vulnerability record updated

vulnerability_report.delete

Vulnerability report deleted

vulnerability_report.update

Vulnerability report updated

workload.create

Workload created

workload.delete

Workload deleted

workload.online

Workload online

workload.recalc_rules

Workload policy recalculated

workload.redetect_network

Workload network redetected

workload.undelete

Workload undeleted

workload.update

Workload settings updated

workload.upgrade

Workload upgraded

workload_interface.create

Workload interface created

workload_interface.delete

Workload interface deleted

workload_interface.update

Workload interface updated

workload_interfaces.update

Workload interfaces updated

For example, IP address changes, new interface added, and interface shut down.

workload_service_report.update

Workload service report updated

workload_settings.update

Workload settings updated

workloads.apply_policy

Workloads policies applied

workloads.bulk_create

Workloads created in bulk

workloads.bulk_delete

Workloads deleted in bulk

workloads.bulk_update

Workloads updated in bulk

workloads.remove_labels

Workloads labels removed

workloads.set_flow_reporting_frequency

Workload flow reporting frequency changed

workloads.set_labels

Workload labels applied

workloads.unpair

Workloads unpaired

workloads.update

Workloads updated

Notification Messages in Events

Events can generate a variety of notifications that are appended after the event type:

  • agent.clone_detected

  • agent.fw_state_table_threshold_exceeded

  • agent.missed_heartbeats

  • agent.missing_heartbeats_after_upgrade

  • agent.policy_deploy_failed

  • agent.policy_deploy_succeeded

  • agent.process_failed

  • agent.service_not_available

  • agent.upgrade_requested

  • agent.upgrade_successful

  • agent.upgrade_time_out

  • container_cluster.duplicate_machine_id

  • container_cluster.region_mismatch

  • container_workload.invalid_pairing_config

  • container_workload.not_created

  • database.temp_table_autocleanup_completed

  • database.temp_table_autocleanup_started

  • hard_limit.exceeded

  • pce.application_started

  • pce.application_stopped

  • remote_syslog.reachable

  • remote_syslog.unreachable

  • request.authentication_failed

  • request.authorization_failed

  • request.internal_server_error

  • request.invalid

  • request.service_unavailable

  • request.unknown_server_error

  • sec_policy.restore

  • soft_limit.exceeded

  • system_task.event_pruning_completed

  • system_task.hard_limit_recovery_completed

  • user.csrf_validation_failed

  • user.login_failed

  • user.login_failure_count_exceeded

  • user.login_session_created

  • user.login_session_terminated

  • user.pce_session_created

  • user.pce_session_terminated

  • user.pw_change_failure

  • user.pw_changed

  • user.pw_complexity_not_met

  • user.pw_reset_completed

  • user.pw_reset_requested

  • virtual_service.not_created

  • workload.duplicate_interface_reported

  • workload.nat_rules_present

  • workload.offline_after_ven_goodbye

  • workload.online

  • workload.oob_policy_changes

  • workload.partial_policy_delivered

  • workload.update_mismatched_interfaces

  • workloads.flow_reporting_frequency_updated

Common Criteria Only Events

The following table lists the types of JSON events that are generated and their descriptions.

For each of these events, the CEF/LEEF success or failure events generated are the event name followed by .success or .failure.

For example, the CEF/LEEF success event for agent.update is agent.update.success and the failure event is agent.update.failure.

JSON Event Type

Description

pce.application_started

PCE application started

pce.application_stopped

PCE application stopped

remote_syslog.reachable

Remote syslog destination reachable

remote_syslog.unreachable

Remote syslog destination not reachable

tls_channel.establish

TLS channel established

tls_channel.terminate

TLS channel terminated

View and Export Events

By default, you can view events in the PCE web console or by using the PCE command line. You can then export Organization events using the PCE web console.

View Events in PCE Web Console

By default, the PCE web console shows events that occur in your organization, such as when a workload is paired, if a pairing failed, when a user logs in or logs out, when a user fails to authenticate, and so on.

If you want to see only certain events you can filter by event type to see events that interest you most. You can also search for Organization events by their universally unique identifier (UUID), and filter events by their severity.

You can also export the list of organization events as a CSV file.

To view Organization events:

  1. From the PCE web console menu, choose Troubleshooting > Events.

  2. As the top of the page, you can use the Event Filter to filter the list by event type.

    event-filter1.png

    Note

    In the Events Viewer, the suggested values for the filters are generated from all possible values. For example, the “Generated By” filter shows all users on the system. However, the actual results displayed by that filter might not contain any data.

    VEN Event Not Displayed in PCE Web Console

    The following events related to VENs are not currently viewable in the PCE web console.

    This is a two-column list of event names.

    VEN Events not shown in PCE Web Console

    fw_tampering_revert_failure

    lost_agent

    fw_tampering_reverted

    missing_os_updates

    fw_tampering_subsystem_failure

    pce_incompat_api_version

    invoke_powershell_failure

    pce_incompat_version

    ipsec_conn_state_change

    pce_reachable

    ipsec_conn_state_failure

    pce_unreachable

    ipsec_monitoring_failure

    proc_config_failure

    ipsec_monitoring_started

    proc_envsetup_failure

    ipsec_monitoring_stopped

    proc_init_failure

    ipsec_subsystem_failure

    proc_malloc_failure

    ipsec_subsystem_started

    proc_restart_failure

    ipsec_subsystem_stopped

    proc_started

    refresh_token_failure

    proc_stopped

    refresh_token_success

VEN href Added to Events Information

After the 22.3.0 upgrade, all events created by a VEN includes the VEN href as well as the previously included Agent href. The VEN href can be used to query the VEN API, obtain the workload record, and execute various operations on the VEN from the PCE.

View Events Using PCE Command Line

Run this command at any runlevel to display:

  • The total number of events

  • The average number of events per day

sudo -u ilo-pce illumio-pce-db-management events-db events-db-show

Run this command at any runlevel to display:

  • The amount of disk space used by events

  • The total number of events

sudo -u ilo-pce illumio-pce-db-management events-db disk-usage-show
Export Events Using PCE Web Console

You can export all Organization events, or export a filtered list organization events to a CSV file.

To export events:

  1. From the PCE web console menu, choose Troubleshooting > Events.

    You see a list of events based on the activities performed.

  2. Click Export > Export All to export all Organization events.

  3. To export a filtered list of a events, filter the list and then click Export > Export Filtered to export only the filtered view.

  4. To search for events based on event type, severity, status, timestamp, and who generated them, use the search filter:

    events-filter1.png
  5. For a faster filtering via the browser, use the following field:

    events-filter2.png
Examples of Events

This section presents examples of recorded events in JSON, CEF, and LEEF for various auditing needs.

User Password Update Failed (JSON)

This example event shows a user password change that failed validation. Event type user.update_password shows "status": "failure", and the notification shows that the user's attempted new password did not meet complexity requirements.

{
        "href": "/orgs/1/events/xxxxxxxx-39bd-43f1-a680-cc17c6984925",
        "timestamp": "2018-08-29T22:07:00.978Z",
        "pce_fqdn": "pce1.bigco.com",
        "created_by": {
               "system": {}
        },
        "event_type": "user.update_password",
        "status": "failure",
        "severity": "info",
        "action": {
               "uuid": "xxxxxxxx-a5f7-4975-a2a5-b4dbd8b74493",
               "api_endpoint": "/login/users/password/update",
               "api_method": "PUT",
               "http_status_code": 302,
               "src_ip": "10.3.6.116"
        },
        "resource_changes": [],
        "notifications": [{
               "uuid": "xxxxxxxx-7b8e-4205-a62a-1f070d8a0ee2",
               "notification_type": "user.pw_complexity_not_met",
               "info": null
        }, {
               "uuid": "xxxxxxxx-9721-4971-b613-d15aa67a4ee7",
               "notification_type": "user.pw_change_failure",
               "info": {
                       "reason": "Password must have minimum of 1 new character(s)"
               }
        }],
        "version": 2
}
Resource Updated (JSON)

This example shows the before and after values of a successful update event rule_set.update. The name of the ruleset changed from "before": "rule_set_2" to "after": "rule_set_3".

{ "href": "/orgs/1/events/xxxxxxxx-8033-4f1a-83e9-fde57c425807",
"timestamp": "2018-08-29T22:04:04.733Z",
"pce_fqdn": "pce1.bigco.com",
"created_by": {
"user": {
"href": "/users/1",
"username": "[email protected]"
}
},
"event_type": "rule_set.update",
"status": "success",
"severity": "info",
"action": {
"uuid": "xxxxxxxx-7488-480b-9ef9-0cd2a8496004",
"api_endpoint": "/api/v2/orgs/1/sec_policy/draft/rule_sets/6",
"api_method": "PUT",
"http_status_code": 204,
"src_ip": "10.3.6.116"
},
"resource_changes": [{
"uuid": "xxxxxxxx-1d13-4e5e-8f0b-e0e8bccc44e0",
"resource": {
"rule_set": {
"href": "/orgs/1/sec_policy/draft/rule_sets/6",
"name": "rule_set_3",
"scopes": [
[{
"label": {
"href": "/orgs/1/labels/19",
"key": "app",
"value": "app2"
}
}, {
"label": {
"href": "/orgs/1/labels/20",
"key": "env",
"value": "env2"
}
}, {
"label": {
"href": "/orgs/1/labels/21",
"key": "loc",
"value": "loc2"
}
}]
]
}
},
"changes": {
"name": {
"before": "rule_set_2",
"after": "rule_set_3"
}
},
"change_type": "update"
}],
"notifications": [],
"version": 2
}
Security Rule Created (JSON)

In this example of a successful sec_rule composite event, a new security rule is created. Because this is a creation event, the before values are null.

{ "href": "/orgs/1/events/xxxxxxxx-6d29-4905-ad32-ee863fb63697",
"timestamp": "2018-08-29T21:48:28.954Z",
"pce_fqdn": "pce24.bigco.com",
"created_by": {
"user": {
"href": "/users/1",
"username": "[email protected]"
}
},
"event_type": "sec_rule.create",
"status": "success",
"severity": "info",
"action": {
"uuid": "xxxxxxxx-165b-4e06-aaac-60e4d8b0b9a0",
"api_endpoint": "/api/v2/orgs/1/sec_policy/draft/rule_sets/1/sec_rules",
"api_method": "POST",
"http_status_code": 201,
"src_ip": "10.6.1.156"
},
"resource_changes": [{
"uuid": "9fcf6feb-bf25-4de8-a68a-a50598df4cf6",
"resource": {
"sec_rule": {
"href": "/orgs/1/sec_policy/draft/rule_sets/1/sec_rules/5"
}
},
"changes": {
"rule_list": {
"before": null,
"after": {
"href": "/orgs/1/sec_policy/draft/rule_sets/1"
}
},
"description": {
"before": null,
"after": "WinRM HTTP/HTTPS and RDP"
},
"type": {
"before": null,
"after": "SecRule"
},
"resolve_labels": {
"before": null,
"after": "1010"
},
"providers": {
"created": [{
"source": true,
"actors": "ams"
}]
},
"destinations": {
"created": [{
"source": false,
"actors": "ams"
}, {
"source": false,
"ip_list": {
"href": "/orgs/1/sec_policy/draft/ip_lists/1"
}
}]
},
"ingress_services": {
"created": [{
"href": "/orgs/1/sec_policy/draft/services/7",
"name": "WinRM HTTP/HTTPS and RDP"
}]
}
},
"change_type": "create"
}],
"notifications": [],
"version": 2
}
User Logged In (JSON)
[
{
  "href": "/orgs/1/events/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
  "timestamp": "2019-06-25T23:34:12.948Z",
  "pce_fqdn": "someFullyQualifiedDomainName",
  "created_by": {
    "user": {
      "href": "/users/1",
      "username": "someUser@someDomain"
    }
  },
  "event_type": "user.sign_in",
  "status": "success",
  "severity": "info",
  "action": {
    "uuid": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
    "api_endpoint": "/login/users/sign_in",
    "api_method": "POST",
    "http_status_code": 302,
    "src_ip": "xxx.xxx.xx.x"
  },
  "resource_changes": [
    {
      "uuid": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
      "resource": {
        "user": {
          "href": "/users/1",
          "type": "local",
          "username": "someUser@someDomain"
        }
      },
      "changes": {
        "sign_in_count": {
          "before": 4,
          "after": 5
        }
      },
      "change_type": "update"
    }
  ],
  "notifications": [
    {
      "uuid": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
      "notification_type": "user.login_session_created",
      "info": {
        "user": {
          "href": "/users/1",
          "type": "local",
          "username": "someUser@someDomain"
        }
      }
    }
  ]
},
{
  "href": "/orgs/1/events/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
  "timestamp": "2019-06-25T23:34:15.147Z",
  "pce_fqdn": "someFullyQualifiedDomainName",
  "created_by": {
    "user": {
      "href": "/users/1",
      "username": "someUser@someDomain"
    }
  },
  "event_type": "user.login",
  "status": "success",
  "severity": "info",
  "action": {
    "uuid": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
    "api_endpoint": "/api/v2/users/login",
    "api_method": "GET",
    "http_status_code": 200,
    "src_ip": "xxx.xxx.xx.x"
  },
  "resource_changes": [
 
  ],
  "notifications": [
    {
      "uuid": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
      "notification_type": "user.pce_session_created",
      "info": {
        "user": {
          "href": "/users/1",
          "username": "someUser@someDomain"
        }
      }
    }
  ]
}
]
User Logged Out (JSON)
[
{
  "href": "/orgs/1/events/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
  "timestamp": "2019-06-25T23:35:16.636Z",
  "pce_fqdn": "someFullyQualifiedDomainName",
  "created_by": {
    "user": {
      "href": "/users/1",
      "username": "someUser@someDomain"
    }
  },
  "event_type": "user.sign_out",
  "status": "success",
  "severity": "info",
  "action": {
    "uuid": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
    "api_endpoint": "/login/logout",
    "api_method": "GET",
    "http_status_code": 302,
    "src_ip": "xxx.xxx.xx.x"
  },
  "resource_changes": [
 
  ],
  "notifications": [
    {
      "uuid": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
      "notification_type": "user.login_session_terminated",
      "info": {
        "reason": "user_logout",
        "user": {
          "href": "/users/1",
          "username": "someUser@someDomain"
        }
      }
    }
  ]
},
{
  "href": "/orgs/1/events/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
  "timestamp": "2019-06-25T23:35:16.636Z",
  "pce_fqdn": "someFullyQualifiedDomainName",
  "created_by": {
    "user": {
      "href": "/users/1",
      "username": "someUser@someDomain"
    }
  },
  "event_type": "user.sign_out",
  "status": "success",
  "severity": "info",
  "action": {
    "uuid": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
    "api_endpoint": "/login/logout",
    "api_method": "GET",
    "http_status_code": 302,
    "src_ip": "xxx.xxx.xx.x"
  },
  "resource_changes": [
 
  ],
  "notifications": [
    {
      "uuid": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
      "notification_type": "user.login_session_terminated",
      "info": {
        "reason": "user_logout",
        "user": {
          "href": "/users/1",
          "username": "someUser@someDomain"
        }
      }
    }
  ]
}
]
Login Failed — Incorrect Username (JSON)
{
  "href": "/orgs/1/events/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
  "timestamp": "2019-06-25T23:35:41.560Z",
  "pce_fqdn": "someFullyQualifiedDomainName",
  "created_by": {
    "system": {
    }
  },
  "event_type": "user.sign_in",
  "status": "failure",
  "severity": "info",
  "action": {
    "uuid": "someFullyQualifiedDomainName",
    "api_endpoint": "/login/users/sign_in",
    "api_method": "POST",
    "http_status_code": 200,
    "src_ip": "xxx.xxx.xx.x"
  },
  "resource_changes": [
 
  ],
  "notifications": [
    {
      "uuid": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
      "notification_type": "user.login_failed",
      "info": {
        "associated_user": {
          "supplied_username": "invalid_username@someDomain"
        }
      }
    }
  ]
}
Login Failed — Incorrect Password (JSON)
{
  "href": "/orgs/1/events/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
  "timestamp": "2019-06-25T23:35:27.649Z",
  "pce_fqdn": "someFullyQualifiedDomainName",
  "created_by": {
    "system": {
    }
  },
  "event_type": "user.sign_in",
  "status": "failure",
  "severity": "info",
  "action": {
    "uuid": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
    "api_endpoint": "/login/users/sign_in",
    "api_method": "POST",
    "http_status_code": 200,
    "src_ip": "xxx.xxx.xx.x"
  },
  "resource_changes": [
 
  ],
  "notifications": [
    {
      "uuid": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
      "notification_type": "user.login_failed",
      "info": {
        "associated_user": {
          "supplied_username": "someUser@someDomain"
        }
      }
    }
  ]
}
User Log Out (CEF)

This example of an event record in CEF shows a successful user log out.

CEF:0|Illumio|PCE|19.3.0|user.logout.success|User Logout Success|1|rt=Mar 06 2020 
18:38:59.900 +0000 dvchost=mypce.com duser=system dst=10.6.5.4 outcome=success 
cat=audit_events request=/api/v2/users/logout_from_jwt requestMethod=POST reason=204
 cs2= cs2Label=resource_changes cs4=[{"uuid":"b5ba8bf0-7ca8-47fc-870f-6c61ddc1648d",
"notification_type":"user.pce_session_terminated","info":{"reason":"user_logout",
"user":{"href":"/users/1","username":"[email protected]"}}}] cs4Label=notifications 
cn2=2 cn2Label=schema-version cs1Label=event_href cs1=/system_events/
e97bd255-4316-4b5e-a885-5b937f756f17
Workload Security Policy Updated (LEEF)

This example of an event record in LEEF shows a successful update of security policy for a workload's Ethernet interfaces.

LEEF:2.0|Illumio|PCE|18.2.0|interface_status.update.success|src=xx.xxx.xxx.xxx
cat=organizational devTime=someUTCdatetime devTimeFormat=yyyy-mm-dd'T'HH:mm:ss.ttttttZ
sev=1 
usrName=albert.einstein url=/orgs/7/agents/someUUID version=2 pce_fqdn=someFQDN 
created_by={"agent":{"href":"/orgs/7/agents/someUUID","hostname":"someHostname"}} 
action={"uuid":"someUUID",
"api_endpoint":"/api/v6/orgs/7/agents/xxxxxx/interface_statuses/update",
"api_method":"PUT","http_status_code":200,"src_ip":"someIP"} 
resource_changes=[{"uuid":"someUUID",
"resource":{"workload":{"href":"/orgs/7/workloads/someUUID","name":null,
"hostname":"someHostname",
"labels":[{"href":"/orgs/7/labels/xxxxxx","key":"loc","value":"test_place_1"},
{"href":"/orgs/7/labels/xxxxxx","key":"env","value":"test_env_1"},
{"href":"/orgs/7/labels/xxxxxx","key":"app","value":"test_app_1"},
{"href":"/orgs/7/labels/xxxxxx","key":"role","value":"test_access_1"}]}},
"changes":{"workload_interfaces":
{"updated":[{"resource":
{"href":"/orgs/7/workloads/someUUID/interfaces/eth1","name":"eth0","
address":{"family":2,"addr":xxxxxxxxx,"mask_addr":someMask}},
"changes":{"address":{"before":null,"after":
{"family":2,"addr":xxxxxxxxx,"mask_addr":someMask}},
"cidr_block":{"before":null,"after":16},"default_gateway_address":
{"before":null,"after":{"family":2,"addr":someGateway,"mask_addr":someMask}},
"link_state":{"before":"unknown","after":"up"},
"network":{"before":null,"after":{"href":"/orgs/7/networks/xx"}},
"network_detection_mode":{"before":null,"after":"single_private_brn"}}},
{"resource":{"href":"/orgs/7/workloads/someUUID/interfaces/eth1",
"name":"eth1","address":{"family":2,"addr":someAddress,"mask_addr":someMask}},
"changes":{"address":{"before":null,"after":{"family":2,"addr":someAddress,
"mask_addr":someMask}},
"cidr_block":{"before":null,"after":16},"link_state":{"before":"unknown","after":"up"},
"network":{"before":null,"after":{"href":"/orgs/7/networks/xx"}},
"network_detection_mode":{"before":null,"after":"single_private_brn"}}}]}},
"change_type":"update"}] notifications=[] event_href=/orgs/7/events/someUUID
Differences from Previous Releases

The following table indicates which event names changed in the Illumio Core 18.2 release. If you are upgrading from a release prior to 18.2, be sure to use the current event name in your alert monitoring system.

Changed VEN Event Names

This table lists the names of VEN-related events prior to the Illumio Core 18.2 release and the names they were changed to in the 18.2 release.

Old Name Prior to 18.2

New Name as of 18.2

fw_config_change

agent.firewall_config

activation_success

activation_failure

agent.activate

deactivation_success

deactivation_failure

agent.deactivate

Events Monitoring Best Practices

The Illumio Core generates a rich stream of structured messages that provide the following information:

  • Illumio PCE system health

  • Illumio PCE notable activity

  • Illumio VEN notable activity

Illumio Core events are structured and actionable. Using the event data, you can identify the severity, affected systems, and what triggered the event. Illumio Core sends the structured messages using the syslog protocol to remote systems, such as Splunk and QRadar. You can set up your remote systems to automatically process the messages and alert you.

Monitoring Operational Practices

In addition to setting up an automated system, Illumio recommends implementing the following operational practices:

  1. Determine the normal quantity of events from the Illumio Core and monitor the trend for changes; investigate spikes or reductions in the event generation rate.

  2. Implement good operational practices to troubleshoot and investigate alerts, and to recover from events.

  3. Do not monitor Illumio Core events in isolation. Monitor them as part of your overall system. Understanding the events in the context of your overall system activity can provide as much information as the events themselves.

Recommended Events to Monitor

As a best practice, Illumio recommendations you monitor the following events at a minimum.

Events

Description

Program name = Illumio_pce/system_health

Severity = Warning, Error, or Fatal

Provides multiple systems metrics, such as CPU and memory data, for each node in a PCE cluster. The PCE generates these events every minute. The Severity field is particularly important. When system metrics exceed thresholds, the severity changes to warning, error, or fatal.

For more information about the metrics and thresholds, see the PCE Administration Guide.

Recommendation: Monitor system_health messages with a severity of warning or higher and correlate the event with other operational monitoring tools to determine if administrative intervention is required.

event_type="lost_agent.found"

Contains the information necessary to identify workloads with lost agents. A lost agent occurs when the PCE deletes a workload from its database but that workload still has a VEN running on it.

Recommendation: Monitor lost_agent.found events and send alerts in case you need to pair the workloads' VENs with the PCE again.

The property "managed" was added to the following APIs:

GET /api/v2/orgs/:xorg_id/workloads

GET /api/v2/orgs/:xorg_id/workloads/:id

GET /api/v2/orgs/:xorg_id/vens

GET /api/v2/orgs/:xorg_id/vens/:id

This property allows customers to easily filter out managed versus unmanaged workloads.

Possible values: 0 or 1

event_type="system_task.agent_missed_heartbeats_check"

Lists the VENs that missed three heartbeats (usually 15 minutes). Typically, this event precedes the PCE taking the VENs offline to perform internal maintenance.

This event triggers an alert to be sent at 25% of the time configured in the offline timer. For example, if the offline timer is configured to 1 hour, an alert is sent after the VEN has not sent a heartbeat for 15 minutes; if the offline timer is configured to 4 hours, an alert is sent after the VEN hasn't sent a heartbeat for 1 hour.

Recommendation: Monitor these events for high-value workloads because the PCE can take these workloads offline when the VENs miss 12 heartbeats (usually 60 minutes).

event_type="system_task.agent_offline_check"

Lists VENs that the PCE has marked offline, usually because they missed 12 heartbeats. The VENs on these workloads haven't communicated with the PCE for an hour and it removed the workloads from policy.

Recommendation: Monitor these events for high-value workloads because they indicate change in the affected workloads' security posture.

event_type="agent.suspend"

Indicates that the VEN is suspended and no longer protecting the workload. If you did not intentionally run the VEN suspend command on the workload, this event can indicate the workload is under attack.

Recommendation: Monitor these events for high-value workloads.

event_type="agent.tampering"

Indicates tampering of the workload's Illumio managed firewall and that the VEN recovered the firewall. Firewall tampering is one of the first signs that a workload is compromised. During a tampering attempt, the VEN and PCE continue to protect the workload; however, you should investigate the cause of the event.

Recommendation: Monitor these events for high-value workloads.

event_type="agent.update"

Contains the state data that the VEN regularly sends to the PCE. Typically, these events contain routine information; however, the VEN can attach a notice indicating the following issues:

  • Processes not running

  • Policy deployment failure

Recommendation: Monitor agent.update events that include notifications because they indicate workloads that might require administrative intervention.

event_type="rule_set.create"

event_type="rule_set.update"

event_type="rule_sets.delete”

Contains the labels indicating the scope of a draft ruleset. Illumio Core generates these events when you create, update, or delete a draft ruleset. When you include “All Applications,” “All Environments,” or “All Locations” in a ruleset scope, the PCE represents that label type as a null HREF. Ruleset scopes that are overly broad affect a large number of workloads. Draft rulesets do not take effect until they are provisioned.

Recommendation: Monitor these events to pinpoint ruleset scopes that are unintentionally overly broad.

event_type="sec_rule.create"

event_type="sec_rule.update"

event_type="sec_rule.delete"

Contains labels indicating when all workloads affected, all services, or a label/label-group are used as a rule source or destination. Illumio Core generates these events when you create, update, or delete a draft ruleset. The removed or added labels could represent high-value applications or environments.

Recommendation: Monitor these events for high-value labels.

event_type="sec_policy.create"

[NEW in Illumio Core 19.3.0] Contains the workloads_affected field, which includes the number of workloads affected by a policy. Illumio Core generates this event when you provision draft policy that updates the policy on affected workloads. The number of affected workloads could be high or a significant percentage of your managed workloads.

Recommendation: Monitor the workloads_affected field for a high number of affect workloads. If the number exceeds an acceptable threshold, investigate the associated the policy.

event_type="agent.clone_detected"

The PCE detects cloned VENs based on clone token mismatch. This is a special alert from the Illumio Core release 19.3.2 onwards, as clones have become a higher priority. Volume of these events make the severity level important and not the fact that these events occurred.

Recommendation: If severity is 1 or ‘error’, some intervention may be needed.