Events Described
This section describes the concepts and types of PCE events.
Event Types, Syntax, and Record Format
When working with events, it is important to recognize their type, REST API schema, syntax, and record information.
Types of Events
The Illumio Core includes the following general categories of auditable events:
Organizational events: Organizational events are further grouped by their source:
API-related events: Events occurring from a use of the REST API, including the PCE web console
System-related events: Events caused by some system-related occurrence
Traffic events
Anonymized Database Dumps
To troubleshoot customer-reported issues, Illumio Customer Support sometimes requests that you supply an anonymized dump of the PCE database.
To safeguard your organization's privacy, the event information is not included in the anonymized database dump.
REST API Events Schema
The Events schema in JSON is downloadable from this documentation portal in the zipfile of the REST API schemas. From the documentation portal Home page, go to the Develop category > REST API Public Schemas (Archive File).
Event Syntax
The names of recorded auditable events in have the following general syntax:
resource.verb[.success_or_failure]
Where:
resource
is a PCE and VEN object, such as PCEuser
or VENagent
component.verb
describes the action of the event on that resource.In CEF and LEEF formats, the success or failure of the verb is included in the recorded event type. This indicator is not needed in the JSON format.
Events Record Information
The following information is included in a event record, which answers the who, what, where, how, and when:
Type of information | Description |
---|---|
Who |
|
What | The action that triggered the event, including the following data:
|
Where | The target resource of the action, composed of the following data:
|
How | API endpoint, method, HTTP status code, and source IP address of the request. |
When | Timestamp of the event's occurrence. This timestamp is not the time the event was recorded. |
Event Record Structure
Regardless of export format (JSON, CEF, or LEEF), the records and fields for all events share a common structure. This common structure of composite events makes post-processing of event data easier.
Bulk change operations on many resources simultaneously are recorded as individual operations on the resource within a single composite event. Failed attempts to change a configuration, such as incorrect authentication, are also collected.
Common Fields
Field Name | Description |
---|---|
| Unique event identifier; contains a UUID. |
| Exact time that the event occurred in RFC 3339 format with fractional seconds. |
| The fully qualified domain name of the PCE; especially useful for Supercluster deployments or if there are multiple PCEs sending data to the SIEM server. |
| Identifies creator of the event; could be a user, the system, or a workload. |
| Name of the event; for more information, see the List of Event Types table. |
| “Success” or “failure;” if the status is null, the event is for information only and doesn't indicate success or failure. |
| “Informational,” “warning,” or “error” indicating the severity of the event. |
| Schema version for events. |
Events Displayed in PCE Web Console
The PCE web console provides an ongoing log of all Organization events that occur in the PCE. For example, Organization events capture actions such as users logging in and logging out, and failed login attempts; when a system object is created, modified, deleted, or provisioned; when a workload is paired or unpaired; and so on.
From the platform and API perspective, Organization events are referred to internally as auditable_events
and are generated by the auditable_events_service
.
You can use the filter at the top of the page to search for events by type of event, event severity level, and when the event occurred.
Cross-Site Request Forgery Protection
A cross-site request forgery (CSRF) is an attack that involves forcing a victim to send an HTTP request to a target destination without their knowledge or intent in order to perform an action as the victim. The underlying cause is an application functionality using predictable URL or form actions in a repeatable way. The nature of the attack is that CSRF exploits the trust that a website has for a user.
For more details on this attack, see the CSRF article on the Web Application Security Consortium website.
Illumio Core can notify you of this type of attack in the following ways:
The PCE web console logs the attack as an Organization Event called “CSRF token validation failure.”
The event is logged in the Illumio Core REST API as
authz_csrf_validation_failure
in theaudit_log_events_get.schema
.The event
authz_csrf_validation_failure
appears in the PCE syslog output if you have deployed the PCE as a software.
Important
When you see this event occur, you should immediately investigate the issue because the request might not have originated from a valid user.
List of Event Types
The following table provides the types of JSON events generated and their description. For each of these events, the CEF/LEEF success or failure events generated are the event name followed by .success
or .failure
.
For example, the CEF/LEEF success event for agent.activate
is agent.activate.success
and the failure event is agent.activate.failure
.
Each event can generate a variety of notification messages. See Notification Messages in Events.
JSON Event Type | Description |
---|---|
| Access restriction created |
| Access restriction deleted |
| Access restriction updated |
| Agent paired |
| Agent clone activated |
| Agent clone detected |
| Agent unpaired |
| Generate maintenance token for any agent |
| Agent disconnected |
| Agent machine identifiers updated |
| Agent refreshed token |
| Policy request sent |
| VEN upgrade request sent |
| Agent reported a service not running |
| Agent suspended |
| Agent firewall tampered |
| Agent unsuspended |
| Agent properties updated. |
| Agent interactive users updated |
| Agent updated existing iptables href |
| Agent updated existing containers |
| Agent existing IP tables uploaded |
| Agent support report uploaded |
| Agent support report request created |
| Agent support report request deleted |
| Condition cleared from a list of VENs |
| Multiple agents unpaired |
| API key created |
| API key deleted |
| API key updated |
| RBAC auth security principal created |
| RBAC auth security principal deleted |
| RBAC auth security principal updated |
| Authentication settings updated |
| PCE cluster created |
| PCE cluster deleted |
| PCE cluster updated |
| Container workload updated |
| Container cluster created |
| Container cluster deleted |
| Container cluster updated |
| Container cluster label mappings updated all at once |
| Container cluster services updated, created, or deleted by Kubelink |
| Container workload profile created |
| Container workload profile deleted |
| Container workload profile updated |
| DB temp table cleanup started |
| DB temp table cleanup completed |
| Domain created |
| Domain deleted |
| Domain updated |
| Enforcement boundary created |
| Enforcement boundary deleted |
| Enforcement boundary updated |
| Event settings updated |
| Global policy settings updated |
| Group created |
| Group updated |
| IP list created |
| IP list deleted |
| IP list updated |
| IP lists deleted |
| IP tables rules created |
| IP tables rules deleted |
| IP tables rules updated |
| Job deleted |
| Label created |
| Label deleted |
| Label updated |
| Label group created |
| Label group deleted |
| Label group updated |
| Labels deleted |
| LDAP configuration created |
| LDAP configuration deleted |
| LDAP configuration updated |
| LDAP server connection verified |
| License deleted |
| License updated |
| Interservice call to login service to create LDAP config |
| Interservice call to login service to delete LDAP config |
| Interservice call to login service to update LDAP config |
| Interservice call to login service to verify connection to the LDAP server |
| New MSP tenant created |
| MSP tenant deleted |
| MSP tenant updated |
| New managed organization created |
| Managed organization deleted |
| Managed organization updated |
| Lost agent found |
| Network created |
| Network deleted |
| Network updated |
| Enforcement instruction applied to a network device |
| Existing or new unmanaged workload assigned to a network device |
| Network device created |
| Network device deleted |
| Network device updated |
| Enforcement instructions applied to multiple network devices |
| Network endpoint created |
| Network endpoint deleted |
| Network endpoint updated |
| Network enforcement node activated |
| Network enforcement node conditions cleared |
| Network enforcement node deactivated |
| Network enforcement node failed or primary lost connectivity to secondary |
| Network enforcement node did not heartbeat for more than 15 minutes |
| Network enforcement node missed heartbeats check |
| Workload added to network endpoint |
| Network enforcement node acknowledgment of policy |
| Network enforcement node policy requested |
| Network enforcement node reports when switches are not reachable |
| A condition was cleared from a list of network enforcement nodes |
| Network function controller created |
| Network function controller deleted |
| Network function controller virtual servers discovered |
| Network function controller policy status |
| Network function controller SLB state updated |
| Organization created |
| Rules for organization recalculated |
| Organization information updated |
| Pairing profile created |
| Pairing profile pairing key created |
| Pairing profile deleted |
| Pairing profile updated |
| Pairing keys deleted from pairing profile |
| Pairing profiles deleted |
| Password policy created |
| Password policy deleted |
| Password policy updated |
| RBAC permission created |
| RBAC permission deleted |
| RBAC permission updated |
| Create domain RADIUS configuration |
| Delete domain RADIUS configuration |
| Update domain RADIUS configuration |
| Verify RADIUS shared secret |
| API request authentication failed |
| API request authorization failed |
| API request failed due to internal server error |
| API request failed due to unavailable service |
| API request failed due to unknown server error |
| Login resource created |
| Login resource deleted |
| Login resource updated |
| Rule set created |
| Rule set deleted |
| Rule set updated |
| Rule sets deleted |
| SAML assertion destination services updated |
| SAML configuration created |
| SAML configuration deleted |
| Generate a new cert for signing SAML AuthN requests |
| SAML configuration updated |
| SAML Service Provider created |
| SAML Service Provider deleted |
| SAML Service Provider updated |
| Security policy created |
| Pending security policy deleted |
| Security policy restored |
| Security policy rules created |
| Security policy rules deleted |
| Security policy rules updated |
| SecureConnect gateway created |
| SecureConnect gateway deleted |
| SecureConnect gateway updated |
| RBAC security principal created |
| RBAC security principal bulk deleted |
| RBAC security principal bulk updated |
| RBAC security principals bulk created |
| Service created |
| Service deleted |
| Service updated |
| Service account created |
| Service account deleted |
| Service account updated |
| Service binding created |
| Service binding created |
| Service bindings deleted |
| Service binding deleted |
| Services deleted |
| Explorer settings updated |
| Server load balancer created |
| Server load balancer deleted |
| Server load balancer updated |
| Support report uploaded |
| syslog remote destination created |
| syslog remote destination deleted |
| syslog remote destination updated |
| Agent missed heartbeats |
| VEN missing heartbeat after upgrade |
| Agents marked offline |
| VEN self signed certificate housekeeping check |
| VEN settings invalidation error state check |
| VEN uninstall timeout |
| Clear VEN authentication recovery condition |
| Compute policy for unmanaged workloads |
| An expired service account api_key was successfully deleted |
| Delete old cached perspectives |
| Endpoint marked offline |
| Container cluster services provisioned |
| Event pruning completed |
| Stale zone subnets removed |
| Set server synced |
| Deactivated and deleted workloads have been vacuumed |
| Traffic collector setting created |
| Traffic collector setting deleted |
| Traffic collector setting updated |
| Trusted proxy IPs created or updated |
| User invitation accepted |
| User authenticated |
| User created |
| User deleted |
| User invited |
| User logged in |
| User login session terminated |
| User logged |
| User session terminated |
| User password reset |
| User session created |
| User session terminated |
| User information updated |
| User password updated |
| User entered expired password |
| User verified MFA |
| Auth token returned for user authentication on PCE |
| User local profile created |
| User local profile deleted |
| User local profile reinvited |
| User local password updated |
| VEN settings updated |
| VEN software release upgraded |
| VEN software release created |
| VEN software release deleted |
| VEN software release deployed |
| VEN software release updated |
| Default VEN software version set |
| Virtual server created |
| Virtual server created |
| Virtual server updated |
| Virtual service created |
| Virtual service deleted |
| Virtual service updated |
| Virtual services created in bulk |
| Virtual services updated in bulk |
| Vulnerability record created |
| Vulnerability record deleted |
| Vulnerability record updated |
| Vulnerability report deleted |
| Vulnerability report updated |
| Workload created |
| Workload deleted |
| Workload online |
| Workload policy recalculated |
| Workload network redetected |
| Workload undeleted |
| Workload settings updated |
| Workload upgraded |
| Workload interface created |
| Workload interface deleted |
| Workload interface updated |
| Workload interfaces updated For example, IP address changes, new interface added, and interface shut down. |
| Workload service report updated |
| Workload settings updated |
| Workloads policies applied |
| Workloads created in bulk |
| Workloads deleted in bulk |
| Workloads updated in bulk |
| Workloads labels removed |
| Workload flow reporting frequency changed |
| Workload labels applied |
| Workloads unpaired |
| Workloads updated |
Notification Messages in Events
Events can generate a variety of notifications that are appended after the event type:
agent.clone_detected
agent.fw_state_table_threshold_exceeded
agent.missed_heartbeats
agent.missing_heartbeats_after_upgrade
agent.policy_deploy_failed
agent.policy_deploy_succeeded
agent.process_failed
agent.service_not_available
agent.upgrade_requested
agent.upgrade_successful
agent.upgrade_time_out
container_cluster.duplicate_machine_id
container_cluster.region_mismatch
container_workload.invalid_pairing_config
container_workload.not_created
database.temp_table_autocleanup_completed
database.temp_table_autocleanup_started
hard_limit.exceeded
pce.application_started
pce.application_stopped
remote_syslog.reachable
remote_syslog.unreachable
request.authentication_failed
request.authorization_failed
request.internal_server_error
request.invalid
request.service_unavailable
request.unknown_server_error
sec_policy.restore
soft_limit.exceeded
system_task.event_pruning_completed
system_task.hard_limit_recovery_completed
user.csrf_validation_failed
user.login_failed
user.login_failure_count_exceeded
user.login_session_created
user.login_session_terminated
user.pce_session_created
user.pce_session_terminated
user.pw_change_failure
user.pw_changed
user.pw_complexity_not_met
user.pw_reset_completed
user.pw_reset_requested
virtual_service.not_created
workload.duplicate_interface_reported
workload.nat_rules_present
workload.offline_after_ven_goodbye
workload.online
workload.oob_policy_changes
workload.partial_policy_delivered
workload.update_mismatched_interfaces
workloads.flow_reporting_frequency_updated
Common Criteria Only Events
The following table lists the types of JSON events that are generated and their descriptions.
For each of these events, the CEF/LEEF success or failure events generated are the event name followed by .success
or .failure
.
For example, the CEF/LEEF success event for agent.update
is agent.update.success
and the failure event is agent.update.failure
.
JSON Event Type | Description |
---|---|
| PCE application started |
| PCE application stopped |
| Remote syslog destination reachable |
| Remote syslog destination not reachable |
| TLS channel established |
| TLS channel terminated |
View and Export Events
By default, you can view events in the PCE web console or by using the PCE command line. You can then export Organization events using the PCE web console.
View Events in PCE Web Console
By default, the PCE web console shows events that occur in your organization, such as when a workload is paired, if a pairing failed, when a user logs in or logs out, when a user fails to authenticate, and so on.
If you want to see only certain events you can filter by event type to see events that interest you most. You can also search for Organization events by their universally unique identifier (UUID), and filter events by their severity.
You can also export the list of organization events as a CSV file.
To view Organization events:
From the PCE web console menu, choose Troubleshooting > Events.
As the top of the page, you can use the Event Filter to filter the list by event type.
Note
In the Events Viewer, the suggested values for the filters are generated from all possible values. For example, the “Generated By” filter shows all users on the system. However, the actual results displayed by that filter might not contain any data.
VEN Event Not Displayed in PCE Web Console The following events related to VENs are not currently viewable in the PCE web console.
This is a two-column list of event names.
VEN Events not shown in PCE Web Console
fw_tampering_revert_failure
lost_agent
fw_tampering_reverted
missing_os_updates
fw_tampering_subsystem_failure
pce_incompat_api_version
invoke_powershell_failure
pce_incompat_version
ipsec_conn_state_change
pce_reachable
ipsec_conn_state_failure
pce_unreachable
ipsec_monitoring_failure
proc_config_failure
ipsec_monitoring_started
proc_envsetup_failure
ipsec_monitoring_stopped
proc_init_failure
ipsec_subsystem_failure
proc_malloc_failure
ipsec_subsystem_started
proc_restart_failure
ipsec_subsystem_stopped
proc_started
refresh_token_failure
proc_stopped
refresh_token_success
VEN href Added to Events Information
After the 22.3.0 upgrade, all events created by a VEN includes the VEN href as well as the previously included Agent href. The VEN href can be used to query the VEN API, obtain the workload record, and execute various operations on the VEN from the PCE.
View Events Using PCE Command Line
Run this command at any runlevel to display:
The total number of events
The average number of events per day
sudo -u ilo-pce illumio-pce-db-management events-db events-db-show
Run this command at any runlevel to display:
The amount of disk space used by events
The total number of events
sudo -u ilo-pce illumio-pce-db-management events-db disk-usage-show
Export Events Using PCE Web Console
You can export all Organization events, or export a filtered list organization events to a CSV file.
To export events:
From the PCE web console menu, choose Troubleshooting > Events.
You see a list of events based on the activities performed.
Click Export > Export All to export all Organization events.
To export a filtered list of a events, filter the list and then click Export > Export Filtered to export only the filtered view.
To search for events based on event type, severity, status, timestamp, and who generated them, use the search filter:
For a faster filtering via the browser, use the following field:
Examples of Events
This section presents examples of recorded events in JSON, CEF, and LEEF for various auditing needs.
User Password Update Failed (JSON)
This example event shows a user password change that failed validation. Event type user.update_password
shows "status": "failure"
, and the notification shows that the user's attempted new password did not meet complexity requirements.
{ "href": "/orgs/1/events/xxxxxxxx-39bd-43f1-a680-cc17c6984925", "timestamp": "2018-08-29T22:07:00.978Z", "pce_fqdn": "pce1.bigco.com", "created_by": { "system": {} }, "event_type": "user.update_password", "status": "failure", "severity": "info", "action": { "uuid": "xxxxxxxx-a5f7-4975-a2a5-b4dbd8b74493", "api_endpoint": "/login/users/password/update", "api_method": "PUT", "http_status_code": 302, "src_ip": "10.3.6.116" }, "resource_changes": [], "notifications": [{ "uuid": "xxxxxxxx-7b8e-4205-a62a-1f070d8a0ee2", "notification_type": "user.pw_complexity_not_met", "info": null }, { "uuid": "xxxxxxxx-9721-4971-b613-d15aa67a4ee7", "notification_type": "user.pw_change_failure", "info": { "reason": "Password must have minimum of 1 new character(s)" } }], "version": 2 }
Resource Updated (JSON)
This example shows the before and after values of a successful update event rule_set.update
. The name of the ruleset changed from "before": "rule_set_2"
to "after": "rule_set_3"
.
{ "href": "/orgs/1/events/xxxxxxxx-8033-4f1a-83e9-fde57c425807", "timestamp": "2018-08-29T22:04:04.733Z", "pce_fqdn": "pce1.bigco.com", "created_by": { "user": { "href": "/users/1", "username": "[email protected]" } }, "event_type": "rule_set.update", "status": "success", "severity": "info", "action": { "uuid": "xxxxxxxx-7488-480b-9ef9-0cd2a8496004", "api_endpoint": "/api/v2/orgs/1/sec_policy/draft/rule_sets/6", "api_method": "PUT", "http_status_code": 204, "src_ip": "10.3.6.116" }, "resource_changes": [{ "uuid": "xxxxxxxx-1d13-4e5e-8f0b-e0e8bccc44e0", "resource": { "rule_set": { "href": "/orgs/1/sec_policy/draft/rule_sets/6", "name": "rule_set_3", "scopes": [ [{ "label": { "href": "/orgs/1/labels/19", "key": "app", "value": "app2" } }, { "label": { "href": "/orgs/1/labels/20", "key": "env", "value": "env2" } }, { "label": { "href": "/orgs/1/labels/21", "key": "loc", "value": "loc2" } }] ] } }, "changes": { "name": { "before": "rule_set_2", "after": "rule_set_3" } }, "change_type": "update" }], "notifications": [], "version": 2 }
Security Rule Created (JSON)
In this example of a successful sec_rule
composite event, a new security rule is created. Because this is a creation event, the before
values are null
.
{ "href": "/orgs/1/events/xxxxxxxx-6d29-4905-ad32-ee863fb63697", "timestamp": "2018-08-29T21:48:28.954Z", "pce_fqdn": "pce24.bigco.com", "created_by": { "user": { "href": "/users/1", "username": "[email protected]" } }, "event_type": "sec_rule.create", "status": "success", "severity": "info", "action": { "uuid": "xxxxxxxx-165b-4e06-aaac-60e4d8b0b9a0", "api_endpoint": "/api/v2/orgs/1/sec_policy/draft/rule_sets/1/sec_rules", "api_method": "POST", "http_status_code": 201, "src_ip": "10.6.1.156" }, "resource_changes": [{ "uuid": "9fcf6feb-bf25-4de8-a68a-a50598df4cf6", "resource": { "sec_rule": { "href": "/orgs/1/sec_policy/draft/rule_sets/1/sec_rules/5" } }, "changes": { "rule_list": { "before": null, "after": { "href": "/orgs/1/sec_policy/draft/rule_sets/1" } }, "description": { "before": null, "after": "WinRM HTTP/HTTPS and RDP" }, "type": { "before": null, "after": "SecRule" }, "resolve_labels": { "before": null, "after": "1010" }, "providers": { "created": [{ "source": true, "actors": "ams" }] }, "destinations": { "created": [{ "source": false, "actors": "ams" }, { "source": false, "ip_list": { "href": "/orgs/1/sec_policy/draft/ip_lists/1" } }] }, "ingress_services": { "created": [{ "href": "/orgs/1/sec_policy/draft/services/7", "name": "WinRM HTTP/HTTPS and RDP" }] } }, "change_type": "create" }], "notifications": [], "version": 2 }
User Logged In (JSON)
[ { "href": "/orgs/1/events/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx", "timestamp": "2019-06-25T23:34:12.948Z", "pce_fqdn": "someFullyQualifiedDomainName", "created_by": { "user": { "href": "/users/1", "username": "someUser@someDomain" } }, "event_type": "user.sign_in", "status": "success", "severity": "info", "action": { "uuid": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx", "api_endpoint": "/login/users/sign_in", "api_method": "POST", "http_status_code": 302, "src_ip": "xxx.xxx.xx.x" }, "resource_changes": [ { "uuid": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx", "resource": { "user": { "href": "/users/1", "type": "local", "username": "someUser@someDomain" } }, "changes": { "sign_in_count": { "before": 4, "after": 5 } }, "change_type": "update" } ], "notifications": [ { "uuid": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx", "notification_type": "user.login_session_created", "info": { "user": { "href": "/users/1", "type": "local", "username": "someUser@someDomain" } } } ] }, { "href": "/orgs/1/events/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx", "timestamp": "2019-06-25T23:34:15.147Z", "pce_fqdn": "someFullyQualifiedDomainName", "created_by": { "user": { "href": "/users/1", "username": "someUser@someDomain" } }, "event_type": "user.login", "status": "success", "severity": "info", "action": { "uuid": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx", "api_endpoint": "/api/v2/users/login", "api_method": "GET", "http_status_code": 200, "src_ip": "xxx.xxx.xx.x" }, "resource_changes": [ ], "notifications": [ { "uuid": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx", "notification_type": "user.pce_session_created", "info": { "user": { "href": "/users/1", "username": "someUser@someDomain" } } } ] } ]
User Logged Out (JSON)
[ { "href": "/orgs/1/events/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx", "timestamp": "2019-06-25T23:35:16.636Z", "pce_fqdn": "someFullyQualifiedDomainName", "created_by": { "user": { "href": "/users/1", "username": "someUser@someDomain" } }, "event_type": "user.sign_out", "status": "success", "severity": "info", "action": { "uuid": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx", "api_endpoint": "/login/logout", "api_method": "GET", "http_status_code": 302, "src_ip": "xxx.xxx.xx.x" }, "resource_changes": [ ], "notifications": [ { "uuid": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx", "notification_type": "user.login_session_terminated", "info": { "reason": "user_logout", "user": { "href": "/users/1", "username": "someUser@someDomain" } } } ] }, { "href": "/orgs/1/events/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx", "timestamp": "2019-06-25T23:35:16.636Z", "pce_fqdn": "someFullyQualifiedDomainName", "created_by": { "user": { "href": "/users/1", "username": "someUser@someDomain" } }, "event_type": "user.sign_out", "status": "success", "severity": "info", "action": { "uuid": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx", "api_endpoint": "/login/logout", "api_method": "GET", "http_status_code": 302, "src_ip": "xxx.xxx.xx.x" }, "resource_changes": [ ], "notifications": [ { "uuid": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx", "notification_type": "user.login_session_terminated", "info": { "reason": "user_logout", "user": { "href": "/users/1", "username": "someUser@someDomain" } } } ] } ]
Login Failed — Incorrect Username (JSON)
{ "href": "/orgs/1/events/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx", "timestamp": "2019-06-25T23:35:41.560Z", "pce_fqdn": "someFullyQualifiedDomainName", "created_by": { "system": { } }, "event_type": "user.sign_in", "status": "failure", "severity": "info", "action": { "uuid": "someFullyQualifiedDomainName", "api_endpoint": "/login/users/sign_in", "api_method": "POST", "http_status_code": 200, "src_ip": "xxx.xxx.xx.x" }, "resource_changes": [ ], "notifications": [ { "uuid": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx", "notification_type": "user.login_failed", "info": { "associated_user": { "supplied_username": "invalid_username@someDomain" } } } ] }
Login Failed — Incorrect Password (JSON)
{ "href": "/orgs/1/events/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx", "timestamp": "2019-06-25T23:35:27.649Z", "pce_fqdn": "someFullyQualifiedDomainName", "created_by": { "system": { } }, "event_type": "user.sign_in", "status": "failure", "severity": "info", "action": { "uuid": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx", "api_endpoint": "/login/users/sign_in", "api_method": "POST", "http_status_code": 200, "src_ip": "xxx.xxx.xx.x" }, "resource_changes": [ ], "notifications": [ { "uuid": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx", "notification_type": "user.login_failed", "info": { "associated_user": { "supplied_username": "someUser@someDomain" } } } ] }
User Log Out (CEF)
This example of an event record in CEF shows a successful user log out.
CEF:0|Illumio|PCE|19.3.0|user.logout.success|User Logout Success|1|rt=Mar 06 2020 18:38:59.900 +0000 dvchost=mypce.com duser=system dst=10.6.5.4 outcome=success cat=audit_events request=/api/v2/users/logout_from_jwt requestMethod=POST reason=204 cs2= cs2Label=resource_changes cs4=[{"uuid":"b5ba8bf0-7ca8-47fc-870f-6c61ddc1648d", "notification_type":"user.pce_session_terminated","info":{"reason":"user_logout", "user":{"href":"/users/1","username":"[email protected]"}}}] cs4Label=notifications cn2=2 cn2Label=schema-version cs1Label=event_href cs1=/system_events/ e97bd255-4316-4b5e-a885-5b937f756f17
Workload Security Policy Updated (LEEF)
This example of an event record in LEEF shows a successful update of security policy for a workload's Ethernet interfaces.
LEEF:2.0|Illumio|PCE|18.2.0|interface_status.update.success|src=xx.xxx.xxx.xxx cat=organizational devTime=someUTCdatetime devTimeFormat=yyyy-mm-dd'T'HH:mm:ss.ttttttZ sev=1 usrName=albert.einstein url=/orgs/7/agents/someUUID version=2 pce_fqdn=someFQDN created_by={"agent":{"href":"/orgs/7/agents/someUUID","hostname":"someHostname"}} action={"uuid":"someUUID", "api_endpoint":"/api/v6/orgs/7/agents/xxxxxx/interface_statuses/update", "api_method":"PUT","http_status_code":200,"src_ip":"someIP"} resource_changes=[{"uuid":"someUUID", "resource":{"workload":{"href":"/orgs/7/workloads/someUUID","name":null, "hostname":"someHostname", "labels":[{"href":"/orgs/7/labels/xxxxxx","key":"loc","value":"test_place_1"}, {"href":"/orgs/7/labels/xxxxxx","key":"env","value":"test_env_1"}, {"href":"/orgs/7/labels/xxxxxx","key":"app","value":"test_app_1"}, {"href":"/orgs/7/labels/xxxxxx","key":"role","value":"test_access_1"}]}}, "changes":{"workload_interfaces": {"updated":[{"resource": {"href":"/orgs/7/workloads/someUUID/interfaces/eth1","name":"eth0"," address":{"family":2,"addr":xxxxxxxxx,"mask_addr":someMask}}, "changes":{"address":{"before":null,"after": {"family":2,"addr":xxxxxxxxx,"mask_addr":someMask}}, "cidr_block":{"before":null,"after":16},"default_gateway_address": {"before":null,"after":{"family":2,"addr":someGateway,"mask_addr":someMask}}, "link_state":{"before":"unknown","after":"up"}, "network":{"before":null,"after":{"href":"/orgs/7/networks/xx"}}, "network_detection_mode":{"before":null,"after":"single_private_brn"}}}, {"resource":{"href":"/orgs/7/workloads/someUUID/interfaces/eth1", "name":"eth1","address":{"family":2,"addr":someAddress,"mask_addr":someMask}}, "changes":{"address":{"before":null,"after":{"family":2,"addr":someAddress, "mask_addr":someMask}}, "cidr_block":{"before":null,"after":16},"link_state":{"before":"unknown","after":"up"}, "network":{"before":null,"after":{"href":"/orgs/7/networks/xx"}}, "network_detection_mode":{"before":null,"after":"single_private_brn"}}}]}}, "change_type":"update"}] notifications=[] event_href=/orgs/7/events/someUUID
Differences from Previous Releases
The following table indicates which event names changed in the Illumio Core 18.2 release. If you are upgrading from a release prior to 18.2, be sure to use the current event name in your alert monitoring system.
Changed VEN Event Names
This table lists the names of VEN-related events prior to the Illumio Core 18.2 release and the names they were changed to in the 18.2 release.
Old Name Prior to 18.2 | New Name as of 18.2 |
---|---|
|
|
|
|
|
|
Events Monitoring Best Practices
The Illumio Core generates a rich stream of structured messages that provide the following information:
Illumio PCE system health
Illumio PCE notable activity
Illumio VEN notable activity
Illumio Core events are structured and actionable. Using the event data, you can identify the severity, affected systems, and what triggered the event. Illumio Core sends the structured messages using the syslog protocol to remote systems, such as Splunk and QRadar. You can set up your remote systems to automatically process the messages and alert you.
Monitoring Operational Practices
In addition to setting up an automated system, Illumio recommends implementing the following operational practices:
Determine the normal quantity of events from the Illumio Core and monitor the trend for changes; investigate spikes or reductions in the event generation rate.
Implement good operational practices to troubleshoot and investigate alerts, and to recover from events.
Do not monitor Illumio Core events in isolation. Monitor them as part of your overall system. Understanding the events in the context of your overall system activity can provide as much information as the events themselves.
Recommended Events to Monitor
As a best practice, Illumio recommendations you monitor the following events at a minimum.
Events | Description |
---|---|
Program name = Severity = Warning, Error, or Fatal | Provides multiple systems metrics, such as CPU and memory data, for each node in a PCE cluster. The PCE generates these events every minute. The Severity field is particularly important. When system metrics exceed thresholds, the severity changes to warning, error, or fatal. For more information about the metrics and thresholds, see the PCE Administration Guide. Recommendation: Monitor |
| Contains the information necessary to identify workloads with lost agents. A lost agent occurs when the PCE deletes a workload from its database but that workload still has a VEN running on it. Recommendation: Monitor The property "managed" was added to the following APIs:
This property allows customers to easily filter out managed versus unmanaged workloads. Possible values: 0 or 1 |
| Lists the VENs that missed three heartbeats (usually 15 minutes). Typically, this event precedes the PCE taking the VENs offline to perform internal maintenance. This event triggers an alert to be sent at 25% of the time configured in the offline timer. For example, if the offline timer is configured to 1 hour, an alert is sent after the VEN has not sent a heartbeat for 15 minutes; if the offline timer is configured to 4 hours, an alert is sent after the VEN hasn't sent a heartbeat for 1 hour. Recommendation: Monitor these events for high-value workloads because the PCE can take these workloads offline when the VENs miss 12 heartbeats (usually 60 minutes). |
| Lists VENs that the PCE has marked offline, usually because they missed 12 heartbeats. The VENs on these workloads haven't communicated with the PCE for an hour and it removed the workloads from policy. Recommendation: Monitor these events for high-value workloads because they indicate change in the affected workloads' security posture. |
| Indicates that the VEN is suspended and no longer protecting the workload. If you did not intentionally run the VEN suspend command on the workload, this event can indicate the workload is under attack. Recommendation: Monitor these events for high-value workloads. |
| Indicates tampering of the workload's Illumio managed firewall and that the VEN recovered the firewall. Firewall tampering is one of the first signs that a workload is compromised. During a tampering attempt, the VEN and PCE continue to protect the workload; however, you should investigate the cause of the event. Recommendation: Monitor these events for high-value workloads. |
| Contains the state data that the VEN regularly sends to the PCE. Typically, these events contain routine information; however, the VEN can attach a notice indicating the following issues:
Recommendation: Monitor |
| Contains the labels indicating the scope of a draft ruleset. Illumio Core generates these events when you create, update, or delete a draft ruleset. When you include “All Applications,” “All Environments,” or “All Locations” in a ruleset scope, the PCE represents that label type as a null HREF. Ruleset scopes that are overly broad affect a large number of workloads. Draft rulesets do not take effect until they are provisioned. Recommendation: Monitor these events to pinpoint ruleset scopes that are unintentionally overly broad. |
| Contains labels indicating when all workloads affected, all services, or a label/label-group are used as a rule source or destination. Illumio Core generates these events when you create, update, or delete a draft ruleset. The removed or added labels could represent high-value applications or environments. Recommendation: Monitor these events for high-value labels. |
| [NEW in Illumio Core 19.3.0] Contains the Recommendation: Monitor the |
| The PCE detects cloned VENs based on clone token mismatch. This is a special alert from the Illumio Core release 19.3.2 onwards, as clones have become a higher priority. Volume of these events make the severity level important and not the fact that these events occurred. Recommendation: If severity is 1 or ‘error’, some intervention may be needed. |