Events Setup
This section describes PCE settings related to events and how to use them to configure PCE behavior.
Requirements for Events Framework
To use the events framework, ensure that you allocate enough disk space for event data, and be familiar with the disk capacity requirements.
Database Sizing for Events
Disk space for a single event is estimated at an average 1,500 bytes.
Caution
As the number of events increases, the increase in disk space is not a straight line. The projections below are rough estimates. Disk usage can vary in production and depends on the type of messages stored.
Number of Events | Disk Space |
|---|---|
25 million | 38GB |
50 million | 58GB |
Data and Disk Capacity for Events
For Illumio Core Cloud customers, Illumio Operations manages all data and disk capacity requirements and configuration for events; including the default events data retention period, database dumps with and without events data, and disk compacting.
For more information, contact your Illumio Support representative.
Events Preview Runtime Setting
If you participated in the preview of Events in 18.1.0, the preview was enabled by configuring a setting in your PCE runtime_env.yml file.
Warning
Remove preview parameter from runtime_env.yml
Before you upgrade to the latest release, you must remove v2_auditable_events_recording_enabled: true from runtime_env.yml. Otherwise, the upgrade does not succeed.
Removing this preview parameter does not affect the collection of “organization events” records, which continue to be recorded.
To remove the Events preview setting:
Edit the
runtime_env.ymlfile and remove the linev2_auditable_events_recording_enabled:v2_auditable_events_recording_enabled: true
If you are not participating in any other previews, you can also remove the line
enable_preview_features.Save your changes.
Events Settings
The following section describes how to configure the Events Settings in the PCE web console.
Events Are Always Enabled
Events are enabled by default in the PCE and cannot be disabled, in accordance with Common Criteria compliance.
Use the PCE web console to change event-related settings and the PCE runtime_env.yml for traffic flow summaries.
Event Settings in PCE Web Console
From the PCE web console, you can change the following event-related settings:
Event Severity: Sets the severity level of events to record. Only messages at the set severity level and higher are recorded. The default severity is “Informational.”
Retention Period: The system retains event records for a specified number of days; from 1 day to 200 days with the default period being 30 days.
Event Pruning: The system automatically prunes events based on disk usage and the age of events; events older than the retention period are pruned. When pruning is complete, the
system_task.prune_old_log_eventsevent is recorded.Event Format: Sets the message output to one of the three formats. The selected message output format only applies to messages that are sent over syslog to a SIEM. The REST API always returns events in JSON.
JavaScript Object Notation (JSON): The default; accepted by Splunk and QRadar SIEMs
Common Event Format (CEF): Accepted by ArcSight
Log Event Extended Format (LEEF): Accepted by QRadar
Event Severity Levels
Severity | Description |
|---|---|
Emergency | System is unusable |
Alert | Should be corrected immediately |
Critical | Critical conditions |
Error | Error conditions |
Warning | Might indicate that an error will occur if action is not taken |
Notice | Events that are unusual, but not error conditions |
Informational | Normal operational messages that require no action |
Debug | Information useful to developers for debugging the application |
Output Format Change
The output format can be changed in the PCE web console:
JSON (default)
CEF
LEEF
Records are in JSON format until you change to one of the other formats. Then, the new events are recorded in the new format; however, the earlier events are not changed to the selected format and they remain recorded in JSON.
Set Event Retention Values
You can set the event retention values depending on the specific conditions described below.
If you are using a SIEM, such as Splunk as the primary long-term storage for events and traffic in a dynamic environment, consider setting the event retention period to 7 days. On setting it to 7 days, you can use the PCE Troubleshooting or Events Viewer to quickly troubleshoot and diagnose events. The benefit of setting 7 days is that if an issue occurs on a Friday, it can still be diagnosed on the following Monday. A large number of events are generated in a dynamic environment, which increases the data stored (disk space used), backup size, and so on. The period of 7 days provides a good balance between disk usage and the ability to troubleshoot.
Note
A dynamic environment is when applications and infrastructure are subject to frequent changes; for example, usage of APIs, ETL, Containers, and so on.
If you are using a SIEM in a non-dynamic environment, consider setting the event retention period to 30 days. A smaller number of events are generated, and less disk space is used in a non-dynamic environment.
If you are not using a SIEM such as Splunk and the PCE is the primary storage for the events data used for reporting, diagnosis, and troubleshooting, set the event retention period as per the organization's record retention policy, such as 30 days. If you generate quarterly reporting using events, set the event retention period to 90 days.
SIEM | Consideration | Value |
|---|---|---|
Yes: Primary storage for events | If primary storage of events is not on the PCE | 7 days (PCE troubleshooting) 1 day (minimum) |
No: Not primary storage for events | If primary storage of events is on the PCE, consider the organization’s record retention policy as well as the available disk and event growth pattern | 30 days (default) |
No |
| As per your record retention policy 200 days (maximum) |
Not applicable | If events data is not needed for reporting or troubleshooting | 1 day (minimum) |
If disk space availability and event growth projections indicate that the desired retention period cannot be safely supported, consider using a SIEM because the PCE might not store events for the desired period.
Note
Running the illumio-pce-db-management events-db command provides an output of the average number of events and the storage used.
Configure Events Settings in PCE Web Console
From the PCE web console menu, choose Settings > Event Settings to view your current settings.
Click Edit to change the settings.
For Event Severity, select from the following options:
Error
Warning
Informational
For Retention Period, enter the number of days you want to retain data.
For Event Format, select from the following options:
JSON
CEF
LEEF
Click Save once you're done.
Limits on Storage From the Illumio Core 19.3.1 release onwards, the PCE will automatically limit the maximum number of events stored. The limits are set on the volume of events stored locally in the PCE database, so that the events recorded in the database do not fill up the disk. The limit is a percentage of the disk capacity, cumulative for all services that store events on the disk.
Important
To change the default limits, contact Illumio Support.
The configuration limit includes both hard and soft limits.
Soft limit: 20% of disk used by event storage
Aggressive pruning is triggered when the soft limit is reached. However, new events are still recorded while pruning. On the Events list page of the PCE Web Console, the
system_task.prune_old_log_eventsevent is displayed with the "Object creation soft limit exceeded" message and 'Severity: Informational'.
Hard limit: 25% of disk used by event storage.
More aggressive pruning is triggered when the hard limit is reached. New events are not recorded while pruning. On the Events list page of the PCE Web Console, the
system_task.prune_old_log_eventsevent is displayed with the message "Object creation hard limit exceeded" message and 'Severity: Error'. The pruning continues until the soft limit level of 20% is reached. When this occurs, asystem_task.hard_limit_recovery_completedevent occurs, and the PCE starts to behave as it did for the soft limit conditions.
SIEM Integration for Events
For analysis or other needs, event data can be sent using syslog to your own analytics or SIEM systems.
About SIEM Integration
This guide also explains how to configure the PCE to securely transfer PCE event data in the following message formats to some associated SIEM systems:
JavaScript Object Notation (JSON), needed for SIEM applications, such as Splunk®.
Common Event Format (CEF), needed for SIEM applications, such as Micro Focus ArcSight®.
Log Event Extended Format (LEEF), needed for SIEM applications, such as IBM QRadar®.
Syslog Forwarding
The PCE can export logs to syslog. You can also use the PCE's own internal syslog configuration.
Identify Events in Syslog Stream
Event records from the syslog stream are identified by the following string:
"version":2 AND '"href":\s*"/orgs/[0-9]*/events' OR '"href":\s*"/system_events/'
Forward Events to External Syslog Server
The PCE has an internal syslog repository, “Local” where all the events get stored. You can control and configure the relaying of syslog messages from the PCE to multiple external syslog servers.
To configure forwarding to an external syslog server:
From the PCE web console menu, choose Settings > Event Settings.
Click Add.
The Event Settings - Add Event Forwarding page opens.
Click Add Repository.
In the Add Repository dialog:
Description: Enter name of the syslog server.
Address: Enter the IP address for the syslog server.
Protocol: Select TCP or UDP. If you select UDP, you only need to enter the port number and click OK to save the configuration.
Port: Enter port number for the syslog server.
TLS: Select Disabled or Enabled. If you select Enabled, click “Choose File” and upload your organization's “Trusted CA Bundle” file from the location it is stored on.
The Trusted CA Bundle contains all the certificates that the PCE (internal syslog service) needs to trust the external syslog server. If you are using a self-signed certificate, that certificate is uploaded. If you are using an internal CA, the certificate of the internal CA must be uploaded as the “Trusted CA Bundle”.
Verify TLS: Select the check-box to ensure that the TLS peer’s server certificate is valid.
Click OK to save the event forwarding configuration.
After ensuring that the events are being forwarded as configured to the correct external syslog servers, you can choose to stop using the “Local” server by editing the local server setting and deselect all message types.
Note
You cannot delete the “Local” server.
Disable Health Check Forwarding
PCE system health messages are useful for PCE operations and monitoring. You can choose to forward them if they are needed on the remote destination.
For example, IBM QRadar is usually used by security personnel, who might not need to monitor the PCE system health. The Illumio App for QRadar does not process the PCE system health messages.
The PCE system health messages are only provided in key/value syslog format. They are not translatable into CEF, LEEF, or JSON formats. If your SIEM does not support processing key/value messages in syslog format, do not forward system health messages to those SIEMs. For example, IBM QRadar and Micro Focus ArcSight do not automatically parse these system health messages.
To disable syslog forwarding of health check messages:
From the PCE web console menu, choose Settings > Event Settings.
Click the Event listed under the Events column.
Under the Events block, for the Status Logs entry, deselect System Health Messages. System health check is only available in key-value format. Selecting a new event format does not change the system health check format to CEF or LEEF.
Click Save.
Note
IBM QRadar and HP ArcSight do not support system health messages. If you are using either of these for SIEM, make sure that you do not select the System Health Messages checkbox.