Syslog Forwarding
The PCE can export logs to syslog. You can also use the PCE's own internal syslog configuration.
The PCE ships with a pre-installed internal (namely, Local) syslog service which is configured and operational by default regardless of network connectivity. For the evaluated configuration, a remote audit server must also be configured so that all PCE audit logs are forwarded to a remote audit server.
Identify Events in Syslog Stream
Event records from the syslog stream are identified by the following string:
"version":2 AND '"href":\s*"/orgs/[0-9]*/events' OR '"href":\s*"/system_events/'
RFC 5424 Message Format Required
Ensure that your remote syslog destination is configured to use the message format defined by RFC 5424, The Syslog Protocol , with the exception.
For a complete listing of the supported PCE audit record types see Appendix A.
Forward Events to External Syslog Server
The PCE has an internal syslog repository, “Local” where all the events get stored. You can control and configure the relaying of syslog messages from the PCE to multiple external syslog servers.
To configure forwarding to an external syslog server:
From the PCE web console menu, choose Settings > Event Settings.
Click Add.
The Event Settings - Add Event Forwarding page opens.
Click Add Repository.
In the Add Repository dialog:
Description: Enter name of the syslog server.
Address: Enter the IP address for the syslog server.
Protocol: Select TCP or UDP. If you select UDP, you only need to enter the port number and click OK to save the configuration.
Port: Enter port number for the syslog server.
TLS: Select Disabled or Enabled. If you select Enabled, click “Choose File” and upload your organization's “Trusted CA Bundle” file from the location it is stored on.
The Trusted CA Bundle contains all the certificates that the PCE (internal syslog service) needs to trust the external syslog server. If you are using a self-signed certificate, that certificate is uploaded. If you are using an internal CA, the certificate of the internal CA must be uploaded as the “Trusted CA Bundle”.
Verify TLS: Select the check-box to ensure that the TLS peer’s server certificate is valid.
Click OK to save the event forwarding configuration.
Note
You cannot delete the “Local” server.
A repository that has been created with TLS “disabled” can be edited to support TLS by clicking on the TLS drop down menu and selecting “Enabled”. Once “Enabled” has been selected, the two related options “Trusted CA Bundle” and “Verify TLS” will appear (See screen shot below):
Figure: Trusted Bundle and Verify TLS
Configuring Remote Audit Server with TLS
For Common Criteria, the communications channel between the PCE and remote syslog destination must be secured by enabling TLS v1.2 as shown above. When adding a new remote syslog repository, a Trusted CA Bundle must be uploaded to the PCE by selecting the certificate bundle configured on the remote syslog server. The PCE TLS client only supports FIPS approved algorithms when communicating with a remote syslog server based on the following cipher suite:
DHE_RSA_WITH_AES_128_GCM_SHA256
If a repository does not have TLS encryption enabled, or the establishment of a TLS connection fails, the Event Configuration page shows a warning icon. Events will not be sent in an unencrypted form.
Figure: Event Settings
Selecting Message Types to Forward
Edit the Local syslog server settings and be sure to select all message types.
From the PCE web console menu, choose Settings > Event Settings.
Click Edit. The Event Settings dialog appears.
Click all the checkboxes for all the event types.
The event types are:
Organizational Events: actions such as users logging in and logging out, and failed login attempts; when a system object is created, modified, deleted, or provisioned; when a workload is paired or unpaired; and so on.
System Events: events that relate to significant activity occurring on the platform that runs the PCE application.
Allowed Traffic Events: events related to traffic that was allowed by the active policy.
Potentially Blocked Traffic Events: events related to traffic that could be blocked; that is, a workload is in a Visibility Only state and the PCE doesn't have rules in the active policy to allow that traffic.
Blocked Traffic Events: Events related to traffic that attempted to communicate with a workload but was blocked due to policy; that is, a workload is in the enforced state and the PCE doesn't have rules in the active policy to allow that traffic.
System Health Messages: Each PCE node reports its status to the local syslog daemon once every minute.
Click Save.
Monitoring for Loss of Forwarded Syslog Messages
(PCE 22.2.30 and later)
The PCE can detect the loss of log messages that should be forwarded to syslog remote destinations. The PCE maintains a queue of log messages to be forwarded. If log messages can not be forwarded to their destination for some reason, the PCE keeps them in the queue and monitors the length of the queue. The status of syslog message forwarding is displayed in the Health page of the Web Console. In the Core Node Health and Data Node Health sections of the PCE Health page, check the line for Syslog Forwarding Status. The possible status messages are Normal (fewer than 5,000 messages in queue), Long message queues (5,000 or more messages in queue), or Dropping messages. When PCE health becomes critical due to loss of the syslog forwarding connection, a message is logged in system_health.log.
Below 5,000 queued messages, the syslog connection state is considered Normal. If the queue size exceeds a threshold of 5000 messages, the connection state changes to Warning. And when messages are dropped for a destination, the connection state changes to Critical.
To set up syslog forwarding monitoring when running in Common Criteria mode, run the following commands on each node:
sudo -u ilo-pce illumio-pce-env metrics syslog_fwd_status:syslog_fwd_status_critical=1 -w sudo -u ilo-pce illumio-pce-ctl restart
The PCE does not do audit log reconciliation when the connection to the syslog server is lost. If the connection between the audit server and the PCE is broken, there may be a gap in the audit server audit record. If a syslog connection is broken, an attempt is made to reconnect to the external syslog destination every 60 seconds.
The following illustration shows the Syslog Forwarding Status when it is Normal:
The following illustration shows the Syslog Forwarding Status when the message queues are getting long:
The following illustration shows the Syslog Forwarding Status when audit messages are being dropped on the data node:
The following illustration shows Syslog Forwarding Status notifications. One of the messages shows how many messages were lost when the syslog connection was lost: "10 messages dropped for repository."
The PCE Administrator can reset the syslog connection statistics by using the following command:
sudo -u ilo-pce illumio-pce-ctl reset-syslog-stats
The underlying cause must also be fixed; otherwise, the status will go back to WARNING or CRITICAL.
Disable Health Check Forwarding
PCE system health messages are useful for PCE operations and monitoring. You can choose to forward them if they are needed on the remote destination.
For example, IBM QRadar is usually used by security personnel, who might not need to monitor the PCE system health. The Illumio App for QRadar does not process the PCE system health messages.
The PCE system health messages are only provided in key/value syslog format. They are not translatable into CEF, LEEF, or JSON formats. If your SIEM does not support processing key/value messages in syslog format, do not forward system health messages to those SIEMs. For example, IBM QRadar and Micro Focus ArcSight do not automatically parse these system health messages.
To disable syslog forwarding of health check messages:
From the PCE web console menu, choose Settings > Event Settings.
Click the Event listed under the Events column.
Under the Events block, for the Status Logs entry, deselect System Health Messages. System health check is only available in key-value format. Selecting a new event format does not change the system health check format to CEF or LEEF.
Click Save.
Note
IBM QRadar and HP ArcSight do not support system health messages. If you are using either of these for SIEM, make sure that you do not select the System Health Messages checkbox.