Events Setup
This section describes PCE settings related to events and how to use them to configure PCE behavior.
Requirements for Events Framework
To use the events framework, ensure that you allocate enough disk space for event data, and be familiar with the disk capacity requirements.
Database Sizing for Events
Disk space for a single event is estimated at an average 1,500 bytes.
Caution
As the number of events increases, the increase in disk space is not a straight line. The projections below are rough estimates. Disk usage can vary in production and depends on the type of messages stored.
Number of Events | Disk Space |
|---|---|
25 million | 38GB |
50 million | 58GB |
Data and Disk Capacity for Events
For Illumio Core Cloud customers, Illumio Operations manages all data and disk capacity requirements and configuration for events; including the default events data retention period, database dumps with and without events data, and disk compacting.
For more information, contact your Illumio Support representative.
For information about the default events data retention period, database dumps with and without events data, disk compacting, and more, see Manage Data and Disk Capacity in the PCE Administration Guide in the Illumio Core.
Events Preview Runtime Setting
If you participated in the preview of Events in 18.1.0, the preview was enabled by configuring a setting in your PCE runtime_env.yml file.
Warning
Remove preview parameter from runtime_env.yml
Before you upgrade to the latest release, you must remove v2_auditable_events_recording_enabled: true from runtime_env.yml. Otherwise, the upgrade does not succeed.
Removing this preview parameter does not affect the collection of “organization events” records, which continue to be recorded.
To remove the Events preview setting:
Edit the
runtime_env.ymlfile and remove the linev2_auditable_events_recording_enabled:v2_auditable_events_recording_enabled: true
If you are not participating in any other previews, you can also remove the line
enable_preview_features.Save your changes.
Events Settings
Events Settings
Configuring Event Audit Levels
The following section describes how to configure the Events Settings in the PCE web console.
Events Are Always Enabled
Events are enabled by default in the PCE and cannot be disabled, in accordance with Common Criteria compliance.
Use the PCE web console to change event-related settings and the PCE runtime_env.yml for traffic flow summaries.
Event Settings in PCE Web Console
From the PCE web console, you can change the following event-related settings:
Event Severity: Sets the severity level of events to record. Only messages at the set severity level and higher are recorded. The default severity is “Informational.”
Retention Period: The system retains event records for a specified number of days; from 1 day to 200 days with the default period being 30 days.
Event Pruning: The system automatically prunes events based on disk usage and the age of events; events older than the retention period are pruned. When pruning is complete, the
system_task.prune_old_log_eventsevent is recorded.Event Format: Sets the message output to one of the three formats. The selected message output format only applies to messages that are sent over syslog to a SIEM. The REST API always returns events in JSON.
JavaScript Object Notation (JSON): The default; accepted by Splunk and QRadar SIEMs
Common Event Format (CEF): Accepted by ArcSight
Log Event Extended Format (LEEF): Accepted by QRadar
Event Severity Levels
Severity | Description |
|---|---|
Emergency | System is unusable |
Alert | Should be corrected immediately |
Critical | Critical conditions |
Error | Error conditions |
Warning | Might indicate that an error will occur if action is not taken |
Notice | Events that are unusual, but not error conditions |
Informational | Normal operational messages that require no action Default audit level for the PCE |
Debug | Information useful to developers for debugging the application |
Output Format Change
The output format can be changed in the PCE web console:
JSON (default)
CEF
LEEF
Records are in JSON format until you change to one of the other formats. Then, the new events are recorded in the new format; however, the earlier events are not changed to the selected format and they remain recorded in JSON.
Set Event Retention Values
You can set the event retention values depending on the specific conditions described below.
If you are using a SIEM, such as Splunk as the primary long-term storage for events and traffic in a dynamic environment, consider setting the event retention period to 7 days. On setting it to 7 days, you can use the PCE Troubleshooting or Events Viewer to quickly troubleshoot and diagnose events. The benefit of setting 7 days is that if an issue occurs on a Friday, it can still be diagnosed on the following Monday. A large number of events are generated in a dynamic environment, which increases the data stored (disk space used), backup size, and so on. The period of 7 days provides a good balance between disk usage and the ability to troubleshoot.
Note
A dynamic environment is when applications and infrastructure are subject to frequent changes; for example, usage of APIs, ETL, Containers, and so on.
If you are using a SIEM in a non-dynamic environment, consider setting the event retention period to 30 days. A smaller number of events are generated, and less disk space is used in a non-dynamic environment.
If you not using a SIEM such as Splunk and the PCE is the primary storage for the events data used for reporting, diagnosis, and troubleshooting, set the event retention period as per the organization's record retention policy, for example 30 days. If you generate quarterly reporting using events, set the event retention period to 90 days.
SIEM | Consideration | Value |
|---|---|---|
Yes: Primary storage for events | If primary storage of events is not on the PCE | 7 days (PCE troubleshooting) 1 day (minimum) |
No: Not primary storage for events | If primary storage of events is on the PCE, consider the organization’s record retention policy as well as the available disk and event growth pattern | 30 days (default) |
No |
| As per your record retention policy 200 days (maximum) |
Not applicable | If events data is not needed for reporting or troubleshooting | 1 day (minimum) |
If disk space availability and event growth projections indicate that the desired retention period cannot be safely supported, consider using a SIEM because the PCE might not store events for the desired period.
Note
Running the illumio-pce-db-management events-db command provides an output of the average number of events and the storage used.
Configure Events Settings in PCE Web Console
From the PCE web console menu, choose Settings > Event Settings to view your current settings.
Click Edit to change the settings.
For Event Severity, select from the following options:
Error
Warning
Informational
For Retention Period, enter the number of days you want to retain data.
For Event Format, select from the following options:
JSON
CEF
LEEF
Click Save once you're done.
Configuring VEN Audit
To configure the PCE to filter VEN audit events based on event type (severity) go the PCE web console main navigation menu and select Settings > Event Configuration. Next, click on Edit and select the event severity from the following list:
Error
Informational
Warning
See “Configure Events Settings in PCE Web Console” for more information.
Limits on Storage
From the Illumio Core 19.3.1 release onwards, the PCE will automatically limit the maximum number of events stored. The limits are set on the volume of events stored locally in the PCE database, so that the events recorded in the database do not fill up the disk. The limit is a percentage of the disk capacity, cumulative for all services that store events on the disk.
Important
To change the default limits, contact Illumio Support.
The configuration limit includes both hard and soft limits.
Soft limit: 20% of disk used by event storage
Aggressive pruning is triggered when the soft limit is reached. However, new events are still recorded while pruning. On the Events list page of the PCE Web Console, the
system_task.prune_old_log_eventsevent is displayed with the "Object creation soft limit exceeded" message and 'Severity: Informational'.
Hard limit: 25% of disk used by event storage.
More aggressive pruning is triggered when the hard limit is reached. New events are not recorded while pruning. On the Events list page of the PCE Web Console, the
system_task.prune_old_log_eventsevent is displayed with the message "Object creation hard limit exceeded" message and 'Severity: Error'. The pruning continues until the soft limit level of 20% is reached. When this occurs, asystem_task.hard_limit_recovery_completedevent occurs, and the PCE starts to behave as it did for the soft limit conditions.
Sync Audit Logs between Local and Remote Syslog Servers
After configuring a new connection for a remote audit server, the PCE automatically resets the local syslog server so that events messages are synced between the local and remote servers. When making a change to the event log settings, it may take a few minutes for the cluster to reflect the updated configuration.
Figure 15: Changes to Event Settings
In the event of a network outage between the remote syslog server and the PCE, there is no log reconciliation between the PCE and remote syslog server.
SIEM Integration for Events
For analysis or other needs, event data can be sent using syslog to your own analytics or SIEM systems.
About SIEM Integration
This guide also explains how to configure the PCE to securely transfer PCE event data in the following message formats to some associated SIEM systems:
JavaScript Object Notation (JSON), needed for SIEM applications, such as Splunk®.
Common Event Format (CEF), needed for SIEM applications, such as Micro Focus ArcSight®.
Log Event Extended Format (LEEF), needed for SIEM applications, such as IBM QRadar®.
Illumio Tools for SIEM Integration
Illumio offers other tools for SIEM integration.
Illumio App for ServiceNow:
Software: Illumio App for CMDB
Documentation: Illumio App for ServiceNow 1.4.0
Syslog Forwarding
The PCE can export logs to syslog. You can also use the PCE's own internal syslog configuration.
The PCE ships with a pre-installed internal (namely, Local) syslog service which is configured and operational by default regardless of network connectivity. For the evaluated configuration, a remote audit server must also be configured so that all PCE audit logs are forwarded to a remote audit server.
Identify Events in Syslog Stream
Event records from the syslog stream are identified by the following string:
"version":2 AND '"href":\s*"/orgs/[0-9]*/events' OR '"href":\s*"/system_events/'
RFC 5424 Message Format Required
Ensure that your remote syslog destination is configured to use the message format defined by RFC 5424, The Syslog Protocol , with the exception.
For a complete listing of the supported PCE audit record types see Appendix A.
Forward Events to External Syslog Server
The PCE has an internal syslog repository, “Local” where all the events get stored. You can control and configure the relaying of syslog messages from the PCE to multiple external syslog servers.
To configure forwarding to an external syslog server:
From the PCE web console menu, choose Settings > Event Settings.
Click Add.
The Event Settings - Add Event Forwarding page opens.
Click Add Repository.
In the Add Repository dialog:
Description: Enter name of the syslog server.
Address: Enter the IP address for the syslog server.
Protocol: Select TCP or UDP. If you select UDP, you only need to enter the port number and click OK to save the configuration.
Port: Enter port number for the syslog server.
TLS: Select Disabled or Enabled. If you select Enabled, click “Choose File” and upload your organization's “Trusted CA Bundle” file from the location it is stored on.
The Trusted CA Bundle contains all the certificates that the PCE (internal syslog service) needs to trust the external syslog server. If you are using a self-signed certificate, that certificate is uploaded. If you are using an internal CA, the certificate of the internal CA must be uploaded as the “Trusted CA Bundle”.
Verify TLS: Select the check-box to ensure that the TLS peer’s server certificate is valid.
Click OK to save the event forwarding configuration.
Note
You cannot delete the “Local” server.
A repository that has been created with TLS “disabled” can be edited to support TLS by clicking on the TLS drop down menu and selecting “Enabled”. Once “Enabled” has been selected, the two related options “Trusted CA Bundle” and “Verify TLS” will appear (See screen shot below):
Figure: Trusted Bundle and Verify TLS
Configuring Remote Audit Server with TLS
For Common Criteria, the communications channel between the PCE and remote syslog destination must be secured by enabling TLS v1.2 as shown above. When adding a new remote syslog repository, a Trusted CA Bundle must be uploaded to the PCE by selecting the certificate bundle configured on the remote syslog server. The PCE TLS client only supports FIPS approved algorithms when communicating with a remote syslog server based on the following cipher suite:
DHE_RSA_WITH_AES_128_GCM_SHA256
If a repository does not have TLS encryption enabled, or the establishment of a TLS connection fails, the Event Configuration page shows a warning icon. Events will not be sent in an unencrypted form.
Figure: Event Settings
Selecting Message Types to Forward
Edit the Local syslog server settings and be sure to select all message types.
From the PCE web console menu, choose Settings > Event Settings.
Click Edit. The Event Settings dialog appears.
Click all the checkboxes for all the event types.
The event types are:
Organizational Events: actions such as users logging in and logging out, and failed login attempts; when a system object is created, modified, deleted, or provisioned; when a workload is paired or unpaired; and so on.
System Events: events that relate to significant activity occurring on the platform that runs the PCE application.
Allowed Traffic Events: events related to traffic that was allowed by the active policy.
Potentially Blocked Traffic Events: events related to traffic that could be blocked; that is, a workload is in a Visibility Only state and the PCE doesn't have rules in the active policy to allow that traffic.
Blocked Traffic Events: Events related to traffic that attempted to communicate with a workload but was blocked due to policy; that is, a workload is in the enforced state and the PCE doesn't have rules in the active policy to allow that traffic.
System Health Messages: Each PCE node reports its status to the local syslog daemon once every minute.
Click Save.
Monitoring for Loss of Forwarded Syslog Messages
(PCE 22.2.30 and later)
The PCE can detect the loss of log messages that should be forwarded to syslog remote destinations. The PCE maintains a queue of log messages to be forwarded. If log messages can not be forwarded to their destination for some reason, the PCE keeps them in the queue and monitors the length of the queue. The status of syslog message forwarding is displayed in the Health page of the Web Console. In the Core Node Health and Data Node Health sections of the PCE Health page, check the line for Syslog Forwarding Status. The possible status messages are Normal (fewer than 5,000 messages in queue), Long message queues (5,000 or more messages in queue), or Dropping messages. When PCE health becomes critical due to loss of the syslog forwarding connection, a message is logged in system_health.log.
Below 5,000 queued messages, the syslog connection state is considered Normal. If the queue size exceeds a threshold of 5000 messages, the connection state changes to Warning. And when messages are dropped for a destination, the connection state changes to Critical.
To set up syslog forwarding monitoring when running in Common Criteria mode, run the following commands on each node:
sudo -u ilo-pce illumio-pce-env metrics syslog_fwd_status:syslog_fwd_status_critical=1 -w sudo -u ilo-pce illumio-pce-ctl restart
The PCE does not do audit log reconciliation when the connection to the syslog server is lost. If the connection between the audit server and the PCE is broken, there may be a gap in the audit server audit record. If a syslog connection is broken, an attempt is made to reconnect to the external syslog destination every 60 seconds.
The following illustration shows the Syslog Forwarding Status when it is Normal:
The following illustration shows the Syslog Forwarding Status when the message queues are getting long:
The following illustration shows the Syslog Forwarding Status when audit messages are being dropped on the data node:
The following illustration shows Syslog Forwarding Status notifications. One of the messages shows how many messages were lost when the syslog connection was lost: "10 messages dropped for repository."
The PCE Administrator can reset the syslog connection statistics by using the following command:
sudo -u ilo-pce illumio-pce-ctl reset-syslog-stats
The underlying cause must also be fixed; otherwise, the status will go back to WARNING or CRITICAL.
Disable Health Check Forwarding
PCE system health messages are useful for PCE operations and monitoring. You can choose to forward them if they are needed on the remote destination.
For example, IBM QRadar is usually used by security personnel, who might not need to monitor the PCE system health. The Illumio App for QRadar does not process the PCE system health messages.
The PCE system health messages are only provided in key/value syslog format. They are not translatable into CEF, LEEF, or JSON formats. If your SIEM does not support processing key/value messages in syslog format, do not forward system health messages to those SIEMs. For example, IBM QRadar and Micro Focus ArcSight do not automatically parse these system health messages.
To disable syslog forwarding of health check messages:
From the PCE web console menu, choose Settings > Event Settings.
Click the Event listed under the Events column.
Under the Events block, for the Status Logs entry, deselect System Health Messages. System health check is only available in key-value format. Selecting a new event format does not change the system health check format to CEF or LEEF.
Click Save.
Note
IBM QRadar and HP ArcSight do not support system health messages. If you are using either of these for SIEM, make sure that you do not select the System Health Messages checkbox.