Events Setup
This section describes PCE settings related to events and how to use them to configure PCE behavior.
Requirements for Events Framework
To use the events framework, allocate enough disk space for event data and be familiar with the disk capacity requirements.
Database Sizing for Events
Disk space for a single event is estimated at an average of 1,500 bytes.
Caution
As the number of events increases, the increase in disk space is not a straight line. The projections below are rough estimates. Disk usage can vary in production and depends on the type of messages stored.
Number of Events | Disk Space |
|---|---|
25 million | 38GB |
50 million | 58GB |
Data and Disk Capacity for Events
For information about the default events data retention period, database dumps with and without events data, disk compacting, and more, see Manage Data and Disk Capacity in the PCE Administration Guide.
Events Preview Runtime Setting
If you participated in the preview of Events in 18.1.0, you enabled it by configuring a setting in your PCE runtime_env.yml file.
Warning
Remove preview parameter from runtime_env.yml
Before you upgrade to the latest release, you must remove v2_auditable_events_recording_enabled:true from runtime_env.yml. Otherwise, the upgrade will not succeed.
Removing this preview parameter does not affect the ongoing recording of “organization events” records.
To remove the Events preview setting:
Edit the
runtime_env.ymlfile and remove the linev2_auditable_events_recording_enabled:v2_auditable_events_recording_enabled: true
If you are not participating in other previews, you can also remove the line
enable_preview_features.Save your changes.
Events Settings
The following section describes how to configure the Events Settings in the PCE web console.
Events Are Always Enabled
Events are enabled by default in the PCE and cannot be disabled by Common Criteria compliance.
Use the PCE web console to change event-related settings and the PCE runtime_env.yml for traffic flow summaries.
Event Settings in PCE Web Console
From the PCE web console, you can change the following event-related settings:
Event Severity: Sets the severity level of events to record. Only messages at the set severity level and higher are recorded. The default severity is “Informational.”
Retention Period: The system retains event records for a specified number of days, ranging from 1 day to 200 days, with a default period of 30 days.
Event Pruning: The system automatically prunes events based on disk usage and their age; events older than the retention period are pruned. When pruning is complete, the
system_task.prune_old_log_eventsevent is recorded.Event Format: This setting sets the message output to one of three formats. The selected message output format only applies to messages sent over Syslog to an SIEM. The REST API always returns events in JSON.
JavaScript Object Notation (JSON): The default; accepted by Splunk and QRadar SIEMs
Common Event Format (CEF): Accepted by ArcSight
Log Event Extended Format (LEEF): Accepted by QRadar
Event Severity Levels
Severity | Description |
|---|---|
Emergency | System is unusable |
Alert | This should be corrected immediately. |
Critical | Critical conditions |
Error | Error conditions |
Warning | Might indicate that an error will occur if action is not taken |
Notice | Unusual events, but not error conditions |
Informational | Normal operational messages that require no action |
Debug | Information useful to developers for debugging the application |
Output Format Change
The output format can be changed in the PCE web console:
JSON (default)
CEF
LEEF
Records are in JSON format until you change to one of the other formats. Then, the new events are recorded in the new format; however, the earlier events are not changed to the selected format and remain recorded in JSON.
Set Event Retention Values
You can set the event retention values depending on the specific conditions described below.
If you use an SIEM, such as Splunk, as the primary long-term storage for events and traffic in a dynamic environment, consider setting the event retention period to 7 days. When setting it to 7 days, you can use the PCE Troubleshooting or Events Viewer to troubleshoot and diagnose events quickly. The benefit of setting it to 7 days is that if an issue occurs on a Friday, it can still be diagnosed the following Monday. Many events are generated in a dynamic environment, increasing the data stored (disk space used), backup size, etc. The period of 7 days provides a good balance between disk usage and the ability to troubleshoot.
Note
A dynamic environment is when applications and infrastructure are subject to frequent changes, such as the usage of APIs, ETL, Containers, and so on.
If you use a SIEM in a non-dynamic environment, consider setting the event retention period to 30 days. In a non-dynamic environment, fewer events are generated, and less disk space is used.
If you are not using a SIEM such as Splunk and the PCE is the primary storage for the events data used for reporting, diagnosis, and troubleshooting, set the event retention period per the organization's record retention policy, such as 30 days. If you generate quarterly reporting using events, set the event retention period to 90 days.
SIEM | Consideration | Value |
|---|---|---|
Yes: Primary storage for events | If the primary storage of events is not on the PCE | 7 days (PCE troubleshooting) 1 day (minimum) |
No: Not primary storage for events | If events are stored primarily on the PCE, consider the organization’s record retention policy and the available disk and event growth pattern. | 30 days (default) |
No |
| As per your record retention policy 200 days (maximum) |
Not applicable | If events data is not needed for reporting or troubleshooting | 1 day (minimum) |
If disk space availability and event growth projections indicate that the desired retention period cannot be safely supported, consider using a SIEM because the PCE might not store events for the desired period.
Note
Running the illumio-pce-db-management events-db command outputs the average number of events and the storage used.
Configure Events Settings in PCE Web Console
From the PCE web console menu, choose Settings > Event Settings to view your current settings.
Click Edit to change the settings.
For Event Severity, select from the following options:
Error
Warning
Informational
For the Retention Period, enter the number of days you want to retain data.
For Event Format, select from the following options:
JSON
CEF
LEEF
Click Save once you're done.
Limits on Storage The PCE will automatically limit the maximum number of events stored. The limits are set on the volume of events stored locally in the PCE database so that the events recorded in the database do not fill the disk. The limit is a percentage of the disk capacity, cumulative for all services that store events on the disk.
Important
To change the default limits, contact Illumio Support.
The configuration limit includes both hard and soft limits.
Soft limit: 20% of disk used by event storage
When the soft limit is reached, aggressive pruning is triggered. However, new events are still recorded while pruning.
On the Events list page of the PCE Web Console, the
system_task.prune_old_log_eventsevent is displayed with the "Object creation soft limit exceeded" message and 'Severity: Informational. '
Hard limit: 25% of disk used by event storage.
More aggressive pruning is triggered when the hard limit is reached. New events are not recorded while pruning.
On the Events list page of the PCE Web Console, the
system_task.prune_old_log_eventsevent is displayed with the message "Object creation hard limit exceeded" and 'Severity: Error'. The pruning continues until the soft limit level of 20% is reached. When this occurs, asystem_task.hard_limit_recovery_completedevent occurs, and the PCE starts to behave as it did for the soft limit conditions.
SIEM Integration for Events
Event data can be sent using syslog to your own analytics or SIEM systems for analysis or other needs.
About SIEM Integration
This guide also explains how to configure the PCE to securely transfer PCE event data in the following message formats to some associated SIEM systems:
JavaScript Object Notation (JSON) is needed for SIEM applications like Splunk®.
Common Event Format (CEF) is needed for Micro Focus ArcSight®.
Log Event Extended Format (LEEF) is needed for IBM QRadar®.
Illumio Tools for SIEM Integration
Illumio offers other tools for SIEM integration.
Illumio App for Splunk:
Software: Technical Add-on for Illumio and Illumio App for Splunk
Documentation: Illumio App for Splunk Guide 4.x
Illumio App for QRadar:
Software: Illumio App for QRadar
Documentation: Illumio App for QRadar Guide 1.4.0
Illumio App for ServiceNow:
Software: Illumio App for CMDB
Documentation: Illumio App for ServiceNow 2.1.0
Syslog Forwarding
The PCE can export logs to syslog. You can also use the PCE's own internal syslog configuration.
Identify Events in Syslog Stream
Event records from the syslog stream are identified by the following string:
"version":2 AND '"href":\s*"/orgs/[0-9]*/events' OR '"href":\s*"/system_events/'
Forward Events to External Syslog Server
The PCE has an internal syslog repository, “Local,” where all the events are stored. You can control and configure the relaying of Syslog messages from the PCE to multiple external Syslog servers.
To configure forwarding to an external Syslog server:
From the PCE web console menu, choose Settings > Event Settings.
Click Add.
The Event Settings - Add Event Forwarding page opens.
Click Add Repository.
In the Add Repository dialog:
Description: Enter the name of the Syslog server.
Address: Enter the IP address for the Syslog server.
Protocol: Select TCP or UDP. If you select UDP, you only need to enter the port number and click OK to save the configuration.
Port: Enter the port number for the syslog server.
TLS: Select Disabled or Enabled. If you select Enabled, click “Choose File” and upload your organization's “Trusted CA Bundle” file from the location where it is stored.
The Trusted CA Bundle contains all the certificates the PCE (internal syslog service) needs to trust the external syslog server. If you are using a self-signed certificate, that certificate is uploaded. If you are using an internal CA, the certificate of the internal CA must be uploaded as the “Trusted CA Bundle”.
Verify TLS: Select the check-box to ensure the TLS peer’s server certificate is valid.
Click OK to save the event forwarding configuration.
After ensuring that the events are being forwarded as configured to the correct external Syslog servers, you can stop using the “Local” server by editing the local server setting and deselecting all message types.
Note
You cannot delete the “Local” server.
Disable Health Check Forwarding
PCE system health messages are helpful for PCE operations and monitoring. If they are needed at the remote destination, you can choose to forward them.
For example, IBM QRadar is usually used by security personnel who might not need to monitor the health of the PCE system. The Illumio App for QRadar does not process the PCE system health messages.
The PCE system health messages are only in key/value syslog format. They are not translatable into CEF, LEEF, or JSON formats. If your SIEM does not support processing key/value messages in Syslog format, do not forward system health messages to those SIEMs. For example, IBM QRadar and Micro Focus ArcSight do not automatically parse these system health messages.
To disable Syslog forwarding of health check messages:
From the PCE web console menu, choose Settings > Event Settings.
Click the Event listed under the Events column.
Under the Events block, for the Status Logs entry, deselect System Health Messages. System health check is only available in key-value format. Selecting a new event format does not change the system health check format to CEF or LEEF.
Click Save.
Note
IBM QRadar and HP ArcSight do not support health messages in the system. If you are using either of these for SIEM, do not select the System Health Messages checkbox.