Events Described
This section describes the concepts and types of PCE events.
List of Event Types
The following table provides the types of JSON events generated and their description. For each of these events, the CEF/LEEF success or failure events generated are the event name followed by .success or .failure.
For example, the CEF/LEEF success event for agent.activate is agent.activate.success and the failure event is agent.activate.failure.
Each event can generate a variety of notification messages. See Notification Messages in Events.
JSON Event Type | Description |
|---|---|
| Access restriction created |
| Access restriction deleted |
| Access restriction updated |
| Agent paired |
| Agent clone activated |
| Agent clone detected |
| Agent unpaired |
| Agent disconnected |
| Agent machine identifiers updated |
| Agent refreshed token |
| Success or failure to apply policy on VEN |
| VEN upgrade requested |
| Agent reported a service not running |
| Agent suspended |
| Agent firewall tampered |
| Agent unsuspended |
| Agent properties updated. |
| Agent interactive users updated |
| Agent updated existing iptables href |
| Agent updated existing containers |
| Agent existing IP tables uploaded |
| Agent support report uploaded |
| Agent support report request created |
| Agent support report request deleted |
| Condition cleared from a list of VENs |
| Multiple agents unpaired |
| API key created |
| API key deleted |
| API key updated |
| RBAC auth security principal created |
| RBAC auth security principal deleted |
| RBAC auth security principal updated |
| Authentication settings updated |
| PCE cluster created |
| PCE cluster deleted |
| PCE cluster updated |
| Container workload updated |
| Container cluster created |
| Container cluster deleted |
| Container cluster updated |
| Container cluster services updated as Kubelink |
| Container workload profile created |
| Container workload profile deleted |
| Container workload profile updated |
| DB temp table cleanup started |
| DB temp table cleanup completed |
| Domain created |
| Domain deleted |
| Domain updated |
| Enforcement boundary created |
| Enforcement boundary deleted |
| Enforcement boundary updated |
| Event settings updated |
| Global policy settings updated |
| Group created |
| Group updated |
| IP list created |
| IP list deleted |
| IP list updated |
| IP lists deleted |
| IP tables rules created |
| IP tables rules deleted |
| IP tables rules updated |
| Job deleted |
| Label created |
| Label deleted |
| Label updated |
| Label group created |
| Label group deleted |
| Label group updated |
| Labels deleted |
| LDAP configuration created |
| LDAP configuration deleted |
| LDAP configuration updated |
| LDAP server connection verified |
| License deleted |
| License created or updated |
| Interservice call to login service to create LDAP config |
| Interservice call to login service to delete LDAP config |
| Interservice call to login service to update LDAP config |
| Interservice call to login service to verify connection to the LDAP server |
| User logged out |
| Lost agent found |
| Network created |
| Network delete |
| Network updated |
| Enforcement instruction applied to a network device |
| Existing or new unmanaged workload assigned to a network device |
| Network device created |
| Network device deleted |
| Network device updated |
| Enforcement instructions applied to multiple network devices |
| Network endpoint created |
| Network endpoint deleted |
| Network endpoint updated |
| Network enforcement node activated |
| Network enforcement node conditions cleared |
| Network enforcement node deactivated |
| Workload added to network endpoint |
| Network enforcement node acknowledgment of policy |
| Network enforcement node policy requested |
| Network enforcement node reports when switches are not reachable |
| Network function controller created |
| Network function controller deleted |
| Network function controller virtual servers discovered |
| Network function controller policy status |
| Network function controller SLB state updated |
| Organization created |
| Rules for organization recalculated |
| Organization information updated |
| Pairing profile created |
| Pairing profile pairing key created |
| Pairing profile deleted |
| Pairing profile updated |
| Pairing keys deleted from pairing profile |
| Pairing profiles deleted |
| Password policy created |
| Password policy deleted |
| Password policy updated |
| RBAC permission created |
| RBAC permission deleted |
| RBAC permission updated |
| API request authentication failed |
| API request authorization failed |
| API request failed due to internal server error |
| API request failed due to unavailable service |
| API request failed due to unknown server error |
| Login resource created |
| Login resource deleted |
| Login resource updated |
| Rule set created |
| Rule set deleted |
| Rule set updated |
| Rule sets deleted |
| SAML assertion consumer services updated |
| SAML configuration created |
| SAML configuration deleted |
| SAML configuration updated |
| SAML Service Provider created |
| SAML Service Provider deleted |
| SAML Service Provider updated |
| Security policy created |
| Pending security policy deleted |
| Security policy restored |
| Security policy rules created |
| Security policy rules deleted |
| Security policy rules updated |
| SecureConnect gateway created |
| SecureConnect gateway deleted |
| SecureConnect gateway updated |
| RBAC security principal created |
| RBAC security principal bulk deleted |
| RBAC security principal bulk updated |
| RBAC security principals bulk created |
| Service created |
| Service deleted |
| Service updated |
| Service binding created |
| Service binding created |
| Service bindings deleted |
| Service binding deleted |
| Services deleted |
| Server load balancer created |
| Server load balancer deleted |
| Server load balancer updated |
| Support report requested |
| Deleted a request for a support report |
| Support report added |
| syslog remote destination created |
| syslog remote destination deleted |
| syslog remote destination updated |
| Agent missed heartbeats |
| Agents marked offline |
| Event pruning completed |
| Traffic collector setting created |
| Traffic collector setting deleted |
| Traffic collector setting updated |
| Trusted proxy IPs created or updated |
| User invitation accepted |
| User authenticated |
| User created |
| User deleted |
| User invited |
| User logged in |
| User login session terminated |
| User logged out |
| User session terminated |
| User password reset |
| User session created |
| User session terminated |
| User information updated |
| User password updated |
| User entered expired password |
| User local profile created |
| User local profile deleted |
| Invitation email resent for local user |
| User local password updated |
| VEN settings updated |
| VEN software release upgraded |
| VEN software release created |
| VEN software release deleted |
| VEN software release deployed |
| VEN software release updated |
| Default VEN software version set |
| Virtual server created |
| Virtual server created |
| Virtual server updated |
| Virtual service created |
| Virtual service deleted |
| Virtual service updated |
| Virtual services created in bulk |
| Virtual services updated in bulk |
| Vulnerability record created |
| Vulnerability record deleted |
| Vulnerability record updated |
| Vulnerability report deleted |
| Vulnerability report created or updated |
| Workload created |
| Workload deleted |
| Workload online |
| Workload policy recalculated |
| Workload network redetected |
| Workload undeleted |
| Workload settings updated |
| Workload upgraded |
| Workload interface created |
| Workload interface deleted |
| Workload interface updated |
| Workload interfaces updated For example, IP address changes, new interface added, and interface shut down. |
| Workload service report updated |
| Workload settings updated |
| Workloads policies applied |
| Workloads created in bulk |
| Workloads deleted in bulk |
| Workloads updated in bulk |
| Workloads labels removed |
| Workload flow reporting frequency changed |
| Workload labels applied |
| Workloads unpaired |
| Workloads updated |
Notification Messages in Events
Events can generate a variety of notifications that are appended after the event type:
agent.clone_detectedagent.fw_state_table_threshold_exceededagent.missed_heartbeatsagent.missing_heartbeats_after_upgradeagent.policy_deploy_failedagent.policy_deploy_succeededagent.process_failedagent.service_not_availableagent.upgrade_requestedagent.upgrade_successfulagent.upgrade_time_outcontainer_cluster.duplicate_machine_idcontainer_cluster.region_mismatchcontainer_workload.invalid_pairing_configcontainer_workload.not_createddatabase.temp_table_autocleanup_completeddatabase.temp_table_autocleanup_startedhard_limit.exceededpce.application_startedpce.application_stoppedremote_syslog.reachableremote_syslog.unreachablerequest.authentication_failedrequest.authorization_failedrequest.internal_server_errorrequest.invalidrequest.service_unavailablerequest.unknown_server_errorsec_policy.restoresoft_limit.exceededsystem_task.event_pruning_completedsystem_task.hard_limit_recovery_completeduser.csrf_validation_faileduser.login_faileduser.login_failure_count_exceededuser.login_session_createduser.login_session_terminateduser.pce_session_createduser.pce_session_terminateduser.pw_change_failureuser.pw_changeduser.pw_complexity_not_metuser.pw_reset_completeduser.pw_reset_requestedvirtual_service.not_createdworkload.duplicate_interface_reportedworkload.nat_rules_presentworkload.offline_after_ven_goodbyeworkload.onlineworkload.oob_policy_changesworkload.partial_policy_deliveredworkload.update_mismatched_interfacesworkloads.flow_reporting_frequency_updated
Event Types, Syntax, and Record Format
When working with events, it is important to recognize their type, REST API schema, syntax, and record information.
Types of Events
The Illumio Core includes the following general categories of auditable events:
Organizational events: Organizational events are further grouped by their source:
API-related events: Events occurring from a use of the REST API, including the PCE web console
System-related events: Events caused by some system-related occurrence
Traffic events
Anonymized Database Dumps
To troubleshoot customer-reported issues, Illumio Customer Support sometimes requests that you supply an anonymized dump of the PCE database.
To safeguard your organization's privacy, the event information is not included in the anonymized database dump.
REST API Events Schema
The Events schema in JSON is downloadable from this documentation portal in the zipfile of the REST API schemas. From the documentation portal Home page, go to the Develop category > REST API Public Schemas (Archive File).
Event Syntax
The names of recorded auditable events in have the following general syntax:
resource.verb[.success_or_failure]
Where:
resourceis a PCE and VEN object, such as PCEuseror VENagentcomponent.verbdescribes the action of the event on that resource.In CEF and LEEF formats, the success or failure of the verb is included in the recorded event type. This indicator is not needed in the JSON format.
Events Record Information
The following information is included in an event record, which answers the who, what, where, how, and when:
Type of information | Description |
|---|---|
Who |
|
What | The action that triggered the event, including the following data:
|
Where | The target resource of the action, composed of the following data:
|
How | API endpoint, method, HTTP status code, and source IP address of the request. |
When | Timestamp of the event's occurrence. This timestamp is not the time the event was recorded. |
Event Record Structure
Regardless of export format (JSON, CEF, or LEEF), the records and fields for all events share a common structure. This common structure of composite events makes post-processing of event data easier.
Bulk change operations on many resources simultaneously are recorded as individual operations on the resource within a single composite event. Failed attempts to change a configuration, such as incorrect authentication, are also collected.
Common Fields
Field Name | Description |
|---|---|
| Unique event identifier; contains a UUID. |
| The exact time that the event occurred was in RFC 3339 format, which was fractional seconds. |
| The fully qualified domain name of the PCE; especially useful for Supercluster deployments or if multiple PCEs send data to the SIEM server. |
| Identifies creator of the event; could be a user, the system, or a workload. |
| Name of the event; see the List of Event Types table for more information. |
| “Success” or “failure;” if the status is null, the event is for information only and doesn't indicate success or failure. |
| “Informational,” “warning,” or “error” indicating the severity of the event. |
| Schema version for events. |
Events Displayed in PCE Web Console
The PCE web console provides an ongoing log of all Organization events in the PCE. For example, Organization events capture actions such as users logging in and logging out, and failed login attempts; when a system object is created, modified, deleted, or provisioned; when a workload is paired or unpaired; and so on.
From the platform and API perspective, Organization events are referred to internally as auditable_events and are generated by the auditable_events_service.
You can use the filter at the top of the page to search for events by type of event, event severity level, and when the event occurred.
Cross-Site Request Forgery Protection
A cross-site request forgery (CSRF) is an attack that involves forcing a victim to send an HTTP request to a target destination without their knowledge or intent to act as the victim. The underlying cause is an application functionality using predictable URL or form actions in a repeatable way. The nature of the attack is that CSRF exploits a website's trust for a user.
For more details on this attack, see the CSRF article on the Web Application Security Consortium website.
Illumio Core can notify you of this type of attack in the following ways:
The PCE web console logs the attack as an Organization Event called “CSRF token validation failure.”
The event is logged in the Illumio Core REST API as
authz_csrf_validation_failurein theaudit_log_events_get.schema.The event
authz_csrf_validation_failureappears in the PCE syslog output if you have deployed the PCE as a software.
Important
When you see this event occur, you should immediately investigate the issue because the request might not have originated from a valid user.
View and Export Events
You can view events by default in the PCE web console or the PCE command line. You can then export Organization events using the PCE web console.
View Events in PCE Web Console
By default, the PCE web console shows events in your organization, such as when a workload is paired, if a pairing failed, when a user logs in or logs out, when a user fails to authenticate, and so on.
If you want to see only certain events, you can filter by event type to see those that interest you most. You can also search for Organization events by their universally unique identifier (UUID) and filter events by their severity.
You can also export the list of organization events as a CSV file.
To view Organization events:
From the PCE web console menu, choose Troubleshooting > Events.
You can filter events by several criteria, such as severity, status, timestamp, or who generated them.
Note
The suggested values for the filters are generated from all possible values. For example, the “Generated By” filter shows all users on the system. However, the actual results displayed by that filter might not contain any data.
VEN Event Not Displayed in PCE Web Console The following events related to VENs are not currently viewable in the PCE web console.
This is a two-column list of event names.
VEN Events not shown in PCE Web Console
fw_tampering_revert_failurelost_agentfw_tampering_revertedmissing_os_updatesfw_tampering_subsystem_failurepce_incompat_api_versioninvoke_powershell_failurepce_incompat_versionipsec_conn_state_changepce_reachableipsec_conn_state_failurepce_unreachableipsec_monitoring_failureproc_config_failureipsec_monitoring_startedproc_envsetup_failureipsec_monitoring_stoppedproc_init_failureipsec_subsystem_failureproc_malloc_failureipsec_subsystem_startedproc_restart_failureipsec_subsystem_stoppedproc_startedrefresh_token_failureproc_stoppedrefresh_token_success
View Events Using PCE Command Line
Run this command at any runlevel to display:
The total number of events
The average number of events per day
sudo -u ilo-pce illumio-pce-db-management events-db events-db-show
Run this command at any runlevel to display:
The amount of disk space used by events
The total number of events
sudo -u ilo-pce illumio-pce-db-management events-db disk-usage-show
Export Events Using PCE Web Console
You can export all Organization events or export a filtered list of organization events to a CSV file.
To export events:
From the PCE web console menu, choose Troubleshooting > Events.
You see a list of events based on the activities performed.
Click Export > Export All to export all Organization events.
To export a filtered list of events, filter the list and then click Export > Export Filtered to export only the filtered view.
Use the search filter for events based on event type, severity, status, timestamp, and who generated them.
Examples of Events
This section presents examples of recorded events in JSON, CEF, and LEEF for various auditing needs.
User Password Update Failed (JSON)
This example event shows a user password change that failed validation. Event type user.update_password shows "status": "failure", and the notification shows that the user's attempted new password did not meet complexity requirements.
{
"href": "/orgs/1/events/xxxxxxxx-39bd-43f1-a680-cc17c6984925",
"timestamp": "2018-08-29T22:07:00.978Z",
"pce_fqdn": "pce1.bigco.com",
"created_by": {
"system": {}
},
"event_type": "user.update_password",
"status": "failure",
"severity": "info",
"action": {
"uuid": "xxxxxxxx-a5f7-4975-a2a5-b4dbd8b74493",
"api_endpoint": "/login/users/password/update",
"api_method": "PUT",
"http_status_code": 302,
"src_ip": "10.3.6.116"
},
"resource_changes": [],
"notifications": [{
"uuid": "xxxxxxxx-7b8e-4205-a62a-1f070d8a0ee2",
"notification_type": "user.pw_complexity_not_met",
"info": null
}, {
"uuid": "xxxxxxxx-9721-4971-b613-d15aa67a4ee7",
"notification_type": "user.pw_change_failure",
"info": {
"reason": "Password must have minimum of 1 new character(s)"
}
}],
"version": 2
}
Resource Updated (JSON)
This example shows the before and after values of a successful update event rule_set.update. The name of the ruleset changed from "before": "rule_set_2" to "after": "rule_set_3".
{ "href": "/orgs/1/events/xxxxxxxx-8033-4f1a-83e9-fde57c425807",
"timestamp": "2018-08-29T22:04:04.733Z",
"pce_fqdn": "pce1.bigco.com",
"created_by": {
"user": {
"href": "/users/1",
"username": "[email protected]"
}
},
"event_type": "rule_set.update",
"status": "success",
"severity": "info",
"action": {
"uuid": "xxxxxxxx-7488-480b-9ef9-0cd2a8496004",
"api_endpoint": "/api/v2/orgs/1/sec_policy/draft/rule_sets/6",
"api_method": "PUT",
"http_status_code": 204,
"src_ip": "10.3.6.116"
},
"resource_changes": [{
"uuid": "xxxxxxxx-1d13-4e5e-8f0b-e0e8bccc44e0",
"resource": {
"rule_set": {
"href": "/orgs/1/sec_policy/draft/rule_sets/6",
"name": "rule_set_3",
"scopes": [
[{
"label": {
"href": "/orgs/1/labels/19",
"key": "app",
"value": "app2"
}
}, {
"label": {
"href": "/orgs/1/labels/20",
"key": "env",
"value": "env2"
}
}, {
"label": {
"href": "/orgs/1/labels/21",
"key": "loc",
"value": "loc2"
}
}]
]
}
},
"changes": {
"name": {
"before": "rule_set_2",
"after": "rule_set_3"
}
},
"change_type": "update"
}],
"notifications": [],
"version": 2
}Security Rule Created (JSON)
In this example of a successful sec_rule composite event, a new security rule is created. Because this is a creation event, the before values are null.
{ "href": "/orgs/1/events/xxxxxxxx-6d29-4905-ad32-ee863fb63697",
"timestamp": "2018-08-29T21:48:28.954Z",
"pce_fqdn": "pce24.bigco.com",
"created_by": {
"user": {
"href": "/users/1",
"username": "[email protected]"
}
},
"event_type": "sec_rule.create",
"status": "success",
"severity": "info",
"action": {
"uuid": "xxxxxxxx-165b-4e06-aaac-60e4d8b0b9a0",
"api_endpoint": "/api/v2/orgs/1/sec_policy/draft/rule_sets/1/sec_rules",
"api_method": "POST",
"http_status_code": 201,
"src_ip": "10.6.1.156"
},
"resource_changes": [{
"uuid": "9fcf6feb-bf25-4de8-a68a-a50598df4cf6",
"resource": {
"sec_rule": {
"href": "/orgs/1/sec_policy/draft/rule_sets/1/sec_rules/5"
}
},
"changes": {
"rule_list": {
"before": null,
"after": {
"href": "/orgs/1/sec_policy/draft/rule_sets/1"
}
},
"description": {
"before": null,
"after": "WinRM HTTP/HTTPS and RDP"
},
"type": {
"before": null,
"after": "SecRule"
},
"resolve_labels": {
"before": null,
"after": "1010"
},
"providers": {
"created": [{
"provider": true,
"actors": "ams"
}]
},
"consumers": {
"created": [{
"provider": false,
"actors": "ams"
}, {
"provider": false,
"ip_list": {
"href": "/orgs/1/sec_policy/draft/ip_lists/1"
}
}]
},
"ingress_services": {
"created": [{
"href": "/orgs/1/sec_policy/draft/services/7",
"name": "WinRM HTTP/HTTPS and RDP"
}]
}
},
"change_type": "create"
}],
"notifications": [],
"version": 2
}User Logged In (JSON)
[
{
"href": "/orgs/1/events/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
"timestamp": "2019-06-25T23:34:12.948Z",
"pce_fqdn": "someFullyQualifiedDomainName",
"created_by": {
"user": {
"href": "/users/1",
"username": "someUser@someDomain"
}
},
"event_type": "user.sign_in",
"status": "success",
"severity": "info",
"action": {
"uuid": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
"api_endpoint": "/login/users/sign_in",
"api_method": "POST",
"http_status_code": 302,
"src_ip": "xxx.xxx.xx.x"
},
"resource_changes": [
{
"uuid": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
"resource": {
"user": {
"href": "/users/1",
"type": "local",
"username": "someUser@someDomain"
}
},
"changes": {
"sign_in_count": {
"before": 4,
"after": 5
}
},
"change_type": "update"
}
],
"notifications": [
{
"uuid": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
"notification_type": "user.login_session_created",
"info": {
"user": {
"href": "/users/1",
"type": "local",
"username": "someUser@someDomain"
}
}
}
]
},
{
"href": "/orgs/1/events/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
"timestamp": "2019-06-25T23:34:15.147Z",
"pce_fqdn": "someFullyQualifiedDomainName",
"created_by": {
"user": {
"href": "/users/1",
"username": "someUser@someDomain"
}
},
"event_type": "user.login",
"status": "success",
"severity": "info",
"action": {
"uuid": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
"api_endpoint": "/api/v2/users/login",
"api_method": "GET",
"http_status_code": 200,
"src_ip": "xxx.xxx.xx.x"
},
"resource_changes": [
],
"notifications": [
{
"uuid": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
"notification_type": "user.pce_session_created",
"info": {
"user": {
"href": "/users/1",
"username": "someUser@someDomain"
}
}
}
]
}
]User Logged Out (JSON)
[
{
"href": "/orgs/1/events/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
"timestamp": "2019-06-25T23:35:16.636Z",
"pce_fqdn": "someFullyQualifiedDomainName",
"created_by": {
"user": {
"href": "/users/1",
"username": "someUser@someDomain"
}
},
"event_type": "user.sign_out",
"status": "success",
"severity": "info",
"action": {
"uuid": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
"api_endpoint": "/login/logout",
"api_method": "GET",
"http_status_code": 302,
"src_ip": "xxx.xxx.xx.x"
},
"resource_changes": [
],
"notifications": [
{
"uuid": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
"notification_type": "user.login_session_terminated",
"info": {
"reason": "user_logout",
"user": {
"href": "/users/1",
"username": "someUser@someDomain"
}
}
}
]
},
{
"href": "/orgs/1/events/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
"timestamp": "2019-06-25T23:35:16.636Z",
"pce_fqdn": "someFullyQualifiedDomainName",
"created_by": {
"user": {
"href": "/users/1",
"username": "someUser@someDomain"
}
},
"event_type": "user.sign_out",
"status": "success",
"severity": "info",
"action": {
"uuid": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
"api_endpoint": "/login/logout",
"api_method": "GET",
"http_status_code": 302,
"src_ip": "xxx.xxx.xx.x"
},
"resource_changes": [
],
"notifications": [
{
"uuid": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
"notification_type": "user.login_session_terminated",
"info": {
"reason": "user_logout",
"user": {
"href": "/users/1",
"username": "someUser@someDomain"
}
}
}
]
}
]Login Failed — Incorrect Username (JSON)
{
"href": "/orgs/1/events/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
"timestamp": "2019-06-25T23:35:41.560Z",
"pce_fqdn": "someFullyQualifiedDomainName",
"created_by": {
"system": {
}
},
"event_type": "user.sign_in",
"status": "failure",
"severity": "info",
"action": {
"uuid": "someFullyQualifiedDomainName",
"api_endpoint": "/login/users/sign_in",
"api_method": "POST",
"http_status_code": 200,
"src_ip": "xxx.xxx.xx.x"
},
"resource_changes": [
],
"notifications": [
{
"uuid": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
"notification_type": "user.login_failed",
"info": {
"associated_user": {
"supplied_username": "invalid_username@someDomain"
}
}
}
]
}Login Failed — Incorrect Password (JSON)
{
"href": "/orgs/1/events/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
"timestamp": "2019-06-25T23:35:27.649Z",
"pce_fqdn": "someFullyQualifiedDomainName",
"created_by": {
"system": {
}
},
"event_type": "user.sign_in",
"status": "failure",
"severity": "info",
"action": {
"uuid": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
"api_endpoint": "/login/users/sign_in",
"api_method": "POST",
"http_status_code": 200,
"src_ip": "xxx.xxx.xx.x"
},
"resource_changes": [
],
"notifications": [
{
"uuid": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
"notification_type": "user.login_failed",
"info": {
"associated_user": {
"supplied_username": "someUser@someDomain"
}
}
}
]
}User Log Out (CEF)
This example of an event record in CEF shows a successful user log out.
CEF:0|Illumio|PCE|19.3.0|user.logout.success|User Logout Success|1|rt=Mar 06 2020
18:38:59.900 +0000 dvchost=mypce.com duser=system dst=10.6.5.4 outcome=success
cat=audit_events request=/api/v2/users/logout_from_jwt requestMethod=POST reason=204
cs2= cs2Label=resource_changes cs4=[{"uuid":"b5ba8bf0-7ca8-47fc-870f-6c61ddc1648d",
"notification_type":"user.pce_session_terminated","info":{"reason":"user_logout",
"user":{"href":"/users/1","username":"[email protected]"}}}] cs4Label=notifications
cn2=2 cn2Label=schema-version cs1Label=event_href cs1=/system_events/
e97bd255-4316-4b5e-a885-5b937f756f17Workload Security Policy Updated (LEEF)
This example of an event record in LEEF shows a successful update of security policy for a workload's Ethernet interfaces.
LEEF:2.0|Illumio|PCE|18.2.0|interface_status.update.success|src=xx.xxx.xxx.xxx
cat=organizational devTime=someUTCdatetime devTimeFormat=yyyy-mm-dd'T'HH:mm:ss.ttttttZ
sev=1
usrName=albert.einstein url=/orgs/7/agents/someUUID version=2 pce_fqdn=someFQDN
created_by={"agent":{"href":"/orgs/7/agents/someUUID","hostname":"someHostname"}}
action={"uuid":"someUUID",
"api_endpoint":"/api/v6/orgs/7/agents/xxxxxx/interface_statuses/update",
"api_method":"PUT","http_status_code":200,"src_ip":"someIP"}
resource_changes=[{"uuid":"someUUID",
"resource":{"workload":{"href":"/orgs/7/workloads/someUUID","name":null,
"hostname":"someHostname",
"labels":[{"href":"/orgs/7/labels/xxxxxx","key":"loc","value":"test_place_1"},
{"href":"/orgs/7/labels/xxxxxx","key":"env","value":"test_env_1"},
{"href":"/orgs/7/labels/xxxxxx","key":"app","value":"test_app_1"},
{"href":"/orgs/7/labels/xxxxxx","key":"role","value":"test_access_1"}]}},
"changes":{"workload_interfaces":
{"updated":[{"resource":
{"href":"/orgs/7/workloads/someUUID/interfaces/eth1","name":"eth0","
address":{"family":2,"addr":xxxxxxxxx,"mask_addr":someMask}},
"changes":{"address":{"before":null,"after":
{"family":2,"addr":xxxxxxxxx,"mask_addr":someMask}},
"cidr_block":{"before":null,"after":16},"default_gateway_address":
{"before":null,"after":{"family":2,"addr":someGateway,"mask_addr":someMask}},
"link_state":{"before":"unknown","after":"up"},
"network":{"before":null,"after":{"href":"/orgs/7/networks/xx"}},
"network_detection_mode":{"before":null,"after":"single_private_brn"}}},
{"resource":{"href":"/orgs/7/workloads/someUUID/interfaces/eth1",
"name":"eth1","address":{"family":2,"addr":someAddress,"mask_addr":someMask}},
"changes":{"address":{"before":null,"after":{"family":2,"addr":someAddress,
"mask_addr":someMask}},
"cidr_block":{"before":null,"after":16},"link_state":{"before":"unknown","after":"up"},
"network":{"before":null,"after":{"href":"/orgs/7/networks/xx"}},
"network_detection_mode":{"before":null,"after":"single_private_brn"}}}]}},
"change_type":"update"}] notifications=[] event_href=/orgs/7/events/someUUIDDifferences from Previous Releases
The following table indicates which event names changed in the Illumio Core 18.2 release. If you are upgrading from a release prior to 18.2, be sure to use the current event name in your alert monitoring system.
Changed VEN Event Names
This table lists the names of VEN-related events prior to the Illumio Core 18.2 release and the names they were changed to in the 18.2 release.
Old Name Prior to 18.2 | New Name as of 18.2 |
|---|---|
|
|
|
|
|
|
Events Monitoring Best Practices
The Illumio Core generates a rich stream of structured messages that provide the following information:
Illumio PCE system health
Illumio PCE notable activity
Illumio VEN notable activity
Illumio Core events are structured and actionable. Using the event data, you can identify the severity, affected systems, and what triggered the event. Illumio Core sends the structured messages using the syslog protocol to remote systems, such as Splunk and QRadar. You can set up your remote systems to automatically process the messages and alert you.
Monitoring Operational Practices
In addition to setting up an automated system, Illumio recommends implementing the following operational practices:
Determine the normal quantity of events from the Illumio Core and monitor the trend for changes; investigate spikes or reductions in the event generation rate.
Implement good operational practices to troubleshoot and investigate alerts and to recover from events.
Do not monitor Illumio Core events in isolation. Monitor them as part of your overall system. Understanding the events in the context of your overall system activity can provide as much information as the events themselves.
Recommended Events to Monitor
As a best practice, Illumio recommends you monitor the following events at a minimum.
Events | Description |
|---|---|
Program name = Severity = Warning, Error, or Fatal | Provides multiple systems metrics, such as CPU and memory data, for each node in a PCE cluster. The PCE generates these events every minute. The Severity field is particularly important. When system metrics exceed thresholds, the severity changes to warning, error, or fatal. For more information about the metrics and thresholds, see the PCE Administration Guide. Recommendation: Monitor |
| Contains the information necessary to identify workloads with lost agents. A lost agent occurs when the PCE deletes a workload from its database, but that workload still has a VEN running on it. Recommendation: Monitor |
| Lists the VENs that missed three heartbeats (default: total of 15 minutes). Typically, this event precedes the PCE taking the VENs offline to perform internal maintenance. For Server VENs, this event triggers an alert to be sent at 25% of the time configured in the offline timer. For example, if the offline timer is configured to 1 hour, an alert is sent after the Server VEN has not sent a heartbeat for 15 minutes; if the offline timer is configured to 4 hours, an alert is sent after the Server VEN hasn't sent a heartbeat for 1 hour. Alerts are disabled by default for Endpoint VENs. Recommendation: Monitor these events for high-value workloads. The PCE can take these workloads offline when the VENs miss 12 heartbeats (usually 60 minutes). |
| This event lists VENs that the PCE has marked offline, usually because they missed 12 heartbeats. The VENs on these workloads haven't communicated with the PCE for an hour and it removed the workloads from policy. Recommendation: Monitor these events for high-value workloads because they indicate a change in the affected workloads' security posture. |
| This event indicates that the VEN is suspended and no longer protecting the workload. If you did not intentionally run the VEN suspend command on the workload, this event can indicate the workload is under attack. Recommendation: Monitor these events for high-value workloads. |
| This event indicates tampering with the workload's Illumio-managed firewall and that the VEN recovered the firewall. Firewall tampering is one of the first signs that a workload is compromised. During a tampering attempt, the VEN and PCE continue to protect the workload; however, you should investigate the event's cause. Recommendation: Monitor these events for high-value workloads. |
| Contains the state data that the VEN regularly sends to the PCE. Typically, these events contain routine information; however, the VEN can attach a notice indicating the following issues:
Recommendation: Monitor |
| Contains the labels indicating the scope of a draft ruleset. Illumio Core generates these events when you create, update, or delete a draft ruleset. When you include “All Applications,” “All Environments,” or “All Locations” in a ruleset scope, the PCE represents that label type as a null HREF. Ruleset scopes that are overly broad affect a large number of workloads. Draft rulesets do not take effect until they are provisioned. Recommendation: Monitor these events to pinpoint ruleset scopes that are unintentionally overly broad. |
| These events contain labels indicating when all workloads affected, all services, or a label/label-group are used as a rule provider or consumer. Illumio Core generates these events when you create, update, or delete a draft ruleset. The removed or added labels could represent high-value applications or environments. Recommendation: Monitor these events for high-value labels. |
| [NEW in Illumio Core 19.3.0] It contains the Recommendation: Monitor the |
| The PCE detects cloned VENs based on clone token mismatch. This is a special alert from release 19.3.2 onwards, as clones have become a higher priority. The volume of these events makes the severity level important, not the fact that these events occurred. Recommendation: If severity is 1 or ‘error’, some intervention may be needed. |