Skip to main content

Illumio Core 22.2 Administration Guide

PCE Logs

Most PCE logs are written to syslog, but some are written directly to a file in the directory you specify with the log_dir parameter in the PCE runtime_env.yml file.

Log Files for PCE Services

This table lists the primary PCE services and the log file name or the syslog filter for the service.

PCE Service

Syslog Filter Rule or Log File Name

agent_service

agent_background_worker_service

program("illumio_pce/agent")

agent_traffic_redis_cache

agent_traffic_redis_server

agent_traffic_redis_replica_server

agent_traffic_service

program("illumio_pce/agent_traffic")

auditable_events_service

message('"category":"auditable"');

collector_service

program("illumio_pce/collector");

database_monitor

program("illumio_pce/database_monitor");

database_servicedatabase_slave_service

program("illumio_pce/postgresql");

ev_service

program("EventService");

executor_service

program("illumio_pce/executor");

fileserver_service

program("illumio_pce/fileserver");

fluentd_source_service

program("illumio_pce/fluentd");

ilocron

program("illumio_pce/ilocron");

login_service

program("illumio_pce/login");

memcached

program("illumio_pce/memcached");

node_monitor

program("illumio_pce/system_health");

redis

program("redis");

search_index_service

program("illumio_pce/search_index");

server_load_balancer

program("haxproxy");

Note

HAProxy logs to /dev/log using a datagram socket. When using syslog-ng, you might need to update your syslog-ng configuration to listen on /dev/log on a datagram socket.

service_discovery_service

program("illumio_pce/service_discovery");

program("consul");

web_server

match("nginx;" value("MESSAGE"));

Log Files (Non-syslog)

The following PCE log files are written to the value defined in the log_dir parameter of the runtime configuration file.

  • agent_background_worker_0.log

  • cache_0_master.log

  • consul.log

  • config_manager

  • fileserver.3400.log

  • fluentd-source.log

  • ilo_node_monitor.log

  • nginx_error.log

  • passenger.log

  • pce_error.log

  • pg_listener.log

  • set_server_0_master.log

  • system_history.log

  • thin_agent_traffic.3200.log

  • thin_collector.3100.log

  • thin_login.3300.log

  • thin_search_engine.3500.log

  • tmessenger/compact.log

  • tmessenger/heartbeat.log

  • tmessenger/relay.log

  • traffic_0_master.log

  • traffic_worker_0.log

  • traffic_worker.log

In addition, the PCE software writes system stats to the following two files in the log_dir/systats directory every 10 minutes:

  • perflog

  • app_stats

Caution

Do not delete these files. They contain helpful system and application statistics that can help Illumio Customer Support troubleshoot PCE operational issues.

System Upgrade Log

On each PCE node, the log_dir directory contains a log file called system_history that records the following information:

  • Initial PCE version

  • PCE version upgrades (old version and new version)

  • PCE backups (how many times the PCE software on the node has been backed up)

  • PCE restores with the timestamp of the backup that was restored

  • A timestamp for each log entry indicating when the operation occurred

Example system upgrade log:

2016-09-24 05:04:11.216: Change in PCE software version detected. Previous: 16.6.0-4114, Current: 16.9.0-4121.
2016-09-24 05:04:39.583: Data dump to file '/tmp/illumio_pce_data/db_backup.tar.gzip' started.
2016-09-24 05:04:47.950: Data dump to file '/tmp/illumio_pce_data/db_backup.tar.gzip' completed. MD5 checksum: 02cef311e9657710a1900d8c5deb49d9
Password-related Event Logging

The system records auditable events for the following occurrences:

  • When an Illumio administrator changes the password requirements

  • When users successfully change their passwords as required by password policy

  • When users fail to change their passwords according to required password policy

Search the PCE Log Files

The PCE Support Report search function allows you to search PCE log files (log files written to /var/log/illumio-pce) based on the following criteria:

  • From (timestamp) & To (timestamp): Search between two specific points in time.

  • From (timestamp) & Duration (hours): Search a duration of time starting at a specific point in time.

  • Duration (hours) & To (timestamp): Search for a duration of time up to a specific point in time.

  • Duration (hours) & At (timestamp): Search for something that occurred during a general time frame and gather logs from before and after the event; (timestamp) is the midpoint.

  • From (timestamp) + Search term: Search from a starting time for specific types of information using the standard search terms.

Examples of Searching

The following examples use questions to frame the log search goals and help formulate your searches.

From/To Dates

Question:

“I want to search 12 hours worth of PCE logs starting on February 1, 2020 and ending 12 hours after (from midnight on 2/1 to noon 2/1).”

Search syntax:

sudo -u ilo-pce ./support_report logs from=02/01/2020 to=02/02/2020
sudo -u ilo-pce ./support_report list

Duration/To

Question:

“I want to search for 6 hours worth of PCE logs ending on midnight of February 2, 2020. Effectively from 1800 on February 1 through 0000 on February 2, 2020.”

The default value of hours in a date is midnight.

Search syntax:

sudo -u ilo-pce ./support_report logs duration=6 to=02/02/2020
sudo -u ilo-pce ./support_report list

At/Duration

Explanation: Use the “at” operator in conjunction with the “duration” operator in the following example. To find details for a specific event that occurred at a known time, use “at.” “At” is the approximate time at which an event of interest occurs. The duration is the time range on either side of this timestamp. In this example, duration returns all messages between 10:00:15 and 12:00:15 on February 2, 2020 and “at” narrows the search to a more specific time, in this case, 11:00AM.

Question:

“I want to search a time window between the hours of 10:00AM and 12:00PM.on February 2, 2020, for a specific event that occurred at 11:00AM.”

Search syntax:

sudo -u ilo-pce ./support_report logs at=02/02/2020T11:00:15 duration=2
sudo -u ilo-pce ./support_report list

From + Search Term Included

Question:

“I want to see all PCE logs entries starting from February 2, 2020, until the present that refer to JOB_STORE."

Search syntax:

sudo -u ilo-pce ./support_report from=02/02/2020 include=JOB_STORE
sudo -u ilo-pce ./support_report list

From + Search Term Included and Excluded

Question:

“I want to see all PCE logs entries starting from February 2, 2020, until the present that refer to JOB_STORE and timed_work but for all servers excluding core0.”

Search syntax:

sudo -u ilo-pce ./support_report from=02/02/2020 include=JOB_STORE include=timed_work exclude=core0
sudo -u ilo-pce ./support_report list