Skip to main content

Integrations

Best Practices to Detect Anomalous Activity Using Splunk

This guide describes how to use Illumio Segmentation telemetry in Splunk to detect anomalous activity.

About Best Practices to Detect Anomalous Activity Using Splunk

Learn how to use Illumio telemetry in Splunk to detect anomalous activity. Review recommended searches, alert considerations, threshold best practices, and examples of Illumio events that you can use to monitor changes that may affect visibility, enforcement, or platform security.

These examples focus on two telemetry categories:

  • Traffic telemetry such as allowed and blocked connections, source and destination activity, and port scans.

  • Operational and security telemetry such as VEN unpairing, suspension, duplicates, ignored interfaces, tampering, offline events, failed PCE logins, and PCE intrusion attempts.

These searches are intended as starting points. Tune the searches based on your environment, including workload criticality, expected maintenance activity, business hours, enforcement mode, label model, and normal application behavior.

Implementation recommendations

Use these recommendations when you implement Splunk searches and alerts on those searches:

  • Validate all searches in a non-production Splunk environment using a limited time range before you enable alerts.

  • Tune thresholds using at least 2 to 4 weeks of baseline data.

  • Correlate Illumio events with change-management records, identity-provider logs, endpoint detection and response telemetry, vulnerability data, and cloud or network logs.

  • Avoid adding alerts to every event without context. Some events such as VEN suspension or ignored interfaces may be legitimate during maintenance windows or while troubleshooting.

  • Use severity tiers. Not every anomaly should generate a high-priority security incident.

  • Review event names and fields against the Illumio and Splunk integration version before deploying the change.

Recommended Splunk search categories

Configure Splunk searches related to these anomaly categories:

  • Traffic anomalies

    • High-volume blocked connections

    • Repeated blocked attempts from the same source

    • Repeated blocked attempts to the same destination

    • Port scans

    • Traffic from unexpected sources to sensitive destinations

    • New or rare ports used against critical workloads

  • VEN operational anomalies

    • Unpairing or deactivation of enforced VENs

    • VEN suspension or unsuspension

    • Duplicate VENs

    • Ignored interfaces added to workloads

    • VEN tampering events

    • VEN offline events

  • PCE access and platform security anomalies

    • PCE failed login attempts

    • Repeated failed logins by user or source IP

    • PCE intrusion attempts or suspicious core OS events

    • Administrative activity outside approved maintenance windows

Alert Threshold Best Practices

Use these alert threshold best practices for detecting anomalous activity in Splunk.

Baseline before enabling alerts

Before enabling alerts, establish a baseline for normal activity. Use these recommended baseline periods:

  • A minimum of 2 weeks for stable production environments.

  • A minimum of 4 weeks or more for environments with weekly or monthly batch jobs, patch cycles, or maintenance windows.

  • Separate baselines for production, development, test, and disaster recovery environments.

Use different thresholds for workload criticality

Enforce strict alert thresholds for high-value or sensitive workloads. For example:

  • Critical production workloads: alert on a small number of unusual blocked attempts or administrative changes.

  • Development workloads: use higher thresholds or lower severity unless the behavior is malicious.

  • Internet-facing workloads: tune to reduce noise from expected scanning.

Prefer correlation over single-event alerts where possible

Some events, such as unpairing an enforced VEN or detecting VEN tampering, require immediate alerts. Teams can detect other events more effectively through correlation, such as repeated blocked connections, failed logins, or scan-like behavior.

Recommended alert severity model
  • Critical

    • VEN tampering on a production or critical workload.

    • Enforced VEN unpaired or deactivated without an approved change.

    • PCE intrusion attempt with evidence of successful access or privilege escalation.

  • High

    • Multiple enforced VENs unpaired or suspended in a short time period.

    • Repeated blocked attempts from one source to many destinations.

    • Repeated PCE failed login attempts followed by a successful login.

    • VEN offline for critical workloads beyond the expected heartbeat tolerance.

  • Medium

    • Ignored interface added to a production workload.

    • Duplicate VEN detected.

    • Repeated blocked attempts to sensitive ports.

    • Failed PCE login attempts above baseline.

  • Low

    • Single administrative change on a non-critical workload.

    • Occasional blocked traffic consistent with known background noise.

Threshold tuning recommendations

Use these threshold tuning recommendations for configuring alerts.

  • Blocked connection attempts from one source to many destinations: more than 10 destinations in 5 minutes.

  • Blocked connection attempts from one source to many ports on one destination: more than 10 ports in 5 minutes.

  • Ping sweep behavior: ICMP activity from one source to more than 20 destinations in 5 minutes.

  • PCE failed login attempts: more than 5 failures for one user or source IP in 10 minutes.

  • VEN offline events: alert immediately for critical workloads; otherwise alert when offline persists beyond normal maintenance or reboot windows.

  • Enforced VEN unpairing, VEN tampering, or unexpected suspension: alert on a single event for production or critical workloads.

Traffic Telemetry Use Cases

These traffic telemetry use cases for detecting anomalous activity in Splunk provide a description of the use case and an example recommended Splunk search.

Blocked connection attempts

Recommended Splunk Search

| tstats summariesonly sum(Traffic.count) AS flows, 
    values(Traffic.src_host) AS src_host, 
    values(Traffic.dest_host) AS dest_host, 
    values(Traffic.direction) AS direction, 
    values(Traffic.transport) AS transport, 
    values(Traffic.src_label_pairs) AS src_labels, 
    values(Traffic.dest_label_pairs) AS dest_labels 
    FROM datamodel=Illumio.Traffic 
    WHERE Traffic.pce_fqdn=" <pce_fqdn> " 
        Traffic.org_id="" 
        (Traffic.src_ip="" OR Traffic.src_host="") 
        (Traffic.dest_ip="" OR Traffic.dest_host="") 
        Traffic.src_label_pairs IN () 
        Traffic.dest_label_pairs IN () 
        Traffic.transport="" 
        Traffic.dest_port IN (*) 
        Traffic.action IN (blocked) 
    BY _time Traffic.timestamp Traffic.pce_fqdn Traffic.src_ip Traffic.dest_ip Traffic.dest_port Traffic.action 
| eval Source = if(isnotnull(src_host), src_host, 'Traffic.src_ip') 
| eval Destination = if(isnotnull(dest_host), dest_host, 'Traffic.dest_ip') 
| table Traffic.timestamp, 
    Traffic.pce_fqdn, 
    Source, 
    direction, 
    Destination, 
    Traffic.dest_port, 
    transport, 
    flows, 
    Traffic.action, 
    src_labels, 
    dest_labels 
| rename Traffic.timestamp AS Timestamp, 
    Traffic.pce_fqdn AS "PCE", 
    direction AS Direction, 
    Traffic.dest_port AS "Port", 
    transport AS Protocol, 
    flows AS Flows, 
    Traffic.action AS "Policy Decision", 
    src_labels AS "Source Labels", 
    dest_labels AS "Destination Labels" 
| fillnull value="-" "Source Labels" "Destination Labels" 
| eval Timestamp = strptime(Timestamp, "%Y-%m-%dT%H:%M:%S%Z") 
| fieldformat Timestamp = strftime(Timestamp, "%+") 
| sort -Timestamp 
What to look for

Blocked connection attempts can indicate policy enforcement working as expected, but repeated or unusual blocked traffic may also indicate reconnaissance, lateral movement attempts, misconfigured applications, or unauthorized access attempts.

How this helps spot anomalous activity

A high number of blocked attempts from a single source can suggest that the source is probing the environment or trying to reach unauthorized services. A high number of blocked attempts to a sensitive destination can indicate that the destination is being targeted.

Recommended alert examples
  • Alert when one source generates blocked attempts to many destinations within a short time window.

  • Alert when one source generates blocked attempts to many ports on the same destination.

  • Alert when blocked attempts target sensitive ports such as SSH, RDP, SMB, WinRM, database ports, or administrative interfaces.

  • Alert when blocked attempts involve workloads labeled as production, crown-jewel, regulated, or PCI/HIPAA/SOX-relevant, depending on your labeling model.

Example investigation questions
  • Is the source workload expected to communicate with the destination?

  • Is the destination a sensitive workload or application tier?

  • Did the activity begin after a deployment, patch, or configuration change?

  • Are there related EDR, authentication, DNS, or vulnerability events for the source?

  • Is policy correctly blocking unauthorized traffic, or does the traffic indicate an application dependency that needs review?

Source or destination-based anomalies

Recommended Splunk search

| tstats summariesonly sum(Traffic.count) AS flows, 
    values(Traffic.src_host) AS src_host, 
    values(Traffic.dest_host) AS dest_host, 
    values(Traffic.direction) AS direction, 
    values(Traffic.transport) AS transport, 
    values(Traffic.src_label_pairs) AS src_labels, 
    values(Traffic.dest_label_pairs) AS dest_labels 
    FROM datamodel=Illumio.Traffic 
    WHERE Traffic.pce_fqdn=" <pce_fqdn>” 
        Traffic.org_id="" 
        (Traffic.src_ip="" OR Traffic.src_host="") 
        (Traffic.dest_ip="" OR Traffic.dest_host="") 
        Traffic.src_label_pairs IN () 
        Traffic.dest_label_pairs IN () 
        Traffic.transport="" 
        Traffic.dest_port IN (*) 
        Traffic.action IN (blocked) 
    BY _time Traffic.timestamp Traffic.pce_fqdn Traffic.src_ip Traffic.dest_ip Traffic.dest_port Traffic.action 
| rename Traffic.dest_port AS port 
| eval Port = if(port > 0, port . " ", "") . upper(transport) 
| stats sum(flows) AS Flows, values(port) AS port, values(transport) AS transport BY Port 
| sort -Flows 
What to look for

Alert on unusual behavior from a specific source or against a specific destination. Use this to monitor critical workloads, jump hosts, domain controllers, database servers, payment systems, and other high-value assets.

How this helps spot anomalous activity

Source-based monitoring can identify compromised workloads attempting to move laterally. Destination-based monitoring can identify sensitive systems being probed or attacked.

Recommended alert examples
  • Alert when a non-administrative source attempts to connect to many destinations.

  • Alert when any source attempts blocked connections to a critical destination.

  • Alert when a workload communicates with a new destination or new port that has not been observed during the baseline period.

  • Alert when traffic violates expected application-tier communication patterns, such as user workstations attempting direct database access.

Port scans

Recommended Splunk search

| savedsearch Illumio_PortScan pce_fqdn=<pce_FQDN> org_id=<org_ID> 
| stats count 
What to look for

Port scans often appear as one source attempting to connect to many ports on one or more destinations within a short period of time.

How this helps spot anomalous activity

Port scanning commonly indicates reconnaissance. It may signal that an attacker is looking for exposed services, a vulnerability scanner is misconfigured, or someone is performing unauthorized internal discovery.

Recommended alert examples
  • Alert when one source attempts connections to more than 10 distinct ports on a single destination within 5 minutes.

  • Alert when one source attempts connections to more than 20 distinct ports across multiple destinations within 5 minutes.

  • Raise severity if the ports include administrative or high-risk services such as SSH, RDP, SMB, WinRM, Telnet, database ports, or remote management interfaces.

Example investigation questions
  • Is the source an approved vulnerability scanner?

  • Was a penetration test or discovery scan scheduled?

  • Are the attempted ports consistent with known tools or malware behavior?

  • Did the source also generate authentication failures or EDR alerts?

Operational and Security Log Use Cases

These operational and security log use cases for detecting anomalous activity in Splunk provide a description of the use case and an example recommended Splunk search.

Unpairing enforced VEN

Recommended Splunk search

`illumio_get_index` sourcetype="illumio:pce" event_type=agents.unpair OR event_type=agent.deactivate 
| lookup illumio_workloads_lookup pce_fqdn org_id href as resource_changes.resource.workload.href output enforcement_mode 
| search enforcement_mode=selective OR enforcement_mode=full 
| rename resource_changes.resource.workload.hostname as VEN resource_changes.resource.workload.href as href 
| table timestamp VEN href enforcement_mode event_type pce_fqdn org_id 
What the query detects

This query identifies VENs that were unpaired or deactivated while their associated workload was in selective or full enforcement.

Why this can indicate anomalous activity

Unpairing or deactivating a VEN removes the workload from active Illumio management. If the workload was enforcing policy, this action may reduce or remove segmentation enforcement for that host.

Recommended alerts
  • Alert on any unpairing or deactivation of a production or critical workload in selective or full enforcement.

  • Alert when multiple enforced VENs are unpaired or deactivated within a short time window.

  • Alert when the action occurs outside a maintenance window.

Threshold guidance
  • Critical workloads: alert on one event.

  • Non-critical workloads: alert when more than one event occurs in 15 minutes, or when the activity is outside normal change windows.

Investigation questions
  • Was the action approved through change management?

  • Who initiated the unpair or deactivation?

  • Was the workload later re-paired?

  • Did enforcement coverage change for a sensitive application?

VEN suspension

Recommended Splunk search

`illumio_get_index` sourcetype="illumio:pce" event_type=agent.update status=success resource_changes.changes.status.before=*suspended OR resource_changes.changes.status.after=*suspended  
| eval Workload=mvdedup('resource_changes.resource.agent.hostname')  
| rename resource_changes.changes.status.before as before resource_changes.changes.status.after as after  
| eval Message=case(before="active" AND after="suspended","Mark as suspended from PCE UI",before="suspended" AND after="active","Unsuspended from PCE UI")  
| append 
     [ search illumio_get_index sourcetype="illumio:pce" event_type=agent.*suspend status=success  
     | eval Workload=mvdedup('created_by.ven.hostname')  
     | eval Message=case(event_type="agent.suspend","Suspended from VEN CLI",event_type="agent.unsuspend","Unsuspended from VEN CLI")]  
| sort _time  
| eval Timestamp=strftime('_time',"%c")  
| rename user as Username  
| fillnull value="OS-user" Username  
| table Timestamp Username Workload Message 
What the query detects

This query identifies VEN suspension and unsuspension activity from the PCE UI and the VEN CLI.

Why this can indicate anomalous activity

Suspension may be legitimate during troubleshooting or maintenance, but unexpected suspension can affect expected VEN behavior and may indicate an attempt to reduce visibility or control over a workload.

Recommended alerts
  • Alert when a production or critical workload is suspended.

  • Alert when suspension is initiated from the VEN CLI by an unexpected OS user.

  • Alert when many VENs are suspended in a short period.

  • Alert when suspension occurs outside an approved maintenance window.

Threshold guidance
  • Critical workloads: alert on one suspension event.

  • Non-critical workloads: alert on multiple suspensions within 15-30 minutes, or when no matching change ticket exists.

Investigation questions
  • Was the suspension authorized?

  • Was it initiated from the PCE UI or locally from the VEN CLI?

  • If CLI-driven, who had OS access to the workload?

  • How long did the VEN remain suspended?

VEN Interfaces have been ignored

Recommended Splunk search

index=* sourcetype="illumio:pce" event_type=workload.update resource_changes.changes.ignored_interface_names.after=*  
| rename resource_changes.resource.workload.name as VEN, resource_changes.changes.ignored_interface_names.after as ignored_interface  
| table _time user status event_type VEN ignored_interface
What the query detects

This query identifies workloads where one or more network interfaces were added to the ignored interface list.

Why this can indicate anomalous activity

Ignoring an interface can reduce visibility or policy coverage for traffic associated with that interface. While this may be valid for some management, backup, virtual, tunnel, or container interfaces, unexpected changes can create blind spots.

Recommended alerts
  • Alert when an interface is ignored on a production or critical workload.

  • Alert when a primary interface is ignored.

  • Alert when many workloads have ignored interfaces added in a short period.

  • Alert when the change is made by an unexpected user or automation account.

Threshold guidance
  • Critical workloads: alert on one ignored-interface change.

  • Non-critical workloads: alert when more than five workloads are changed within 30 minutes, or when the acting user is unusual.

Investigation questions
  • Is the ignored interface expected for this workload type?

  • Is the interface used for production, management, backup, tunnel, or container traffic?

  • Did the change reduce visibility into important flows?

  • Should the interface be removed from the ignored list?

VEN duplicates

Recommended Splunk search

index=* sourcetype="illumio:pce" event_type= " agent.clone_detected " 
What to look for

Duplicate VENs can occur when a system image, clone, or template is deployed without properly preparing or re-pairing the VEN identity. Duplicate VENs may also indicate operational process gaps in image management or workload provisioning.

How this helps spot anomalous activity

A duplicate VEN may cause confusion during investigation and may affect visibility, inventory hygiene, and policy operations. In some cases, unexpected duplicates may indicate unauthorized cloning or improper reuse of workload images.

Recommended alerts
  • Alert when duplicate VEN or workload identity events appear.

  • Alert when duplicates involve production or sensitive workloads.

  • Alert when duplicate VENs appear after a burst of provisioning activity.

Threshold guidance
  • Production or sensitive workloads: alert on one duplicate event.

  • Large provisioning environments: alert when duplicates exceed the normal baseline for a deployment window.

Investigation questions
  • Was a workload cloned from a template that already had a paired VEN?

  • Are the duplicate workloads expected from an autoscaling or provisioning process?

  • Which VEN identity should remain active?

  • Does the image build process need to be corrected?

VEN tampering events

Recommended Splunk search

| savedsearch Illumio_Firewall_Tampering 
| search pce_fqdn=<pce_FQDN> org_id=<org_ID> 
| stats count 
What to look for

VEN tampering events may indicate attempts to stop, disable, alter, or interfere with the VEN or its supporting services and files.

How this helps spot anomalous activity

Tampering can indicate an attempt to weaken host-based visibility or segmentation enforcement. Treat this as a high-priority event, especially on production, internet-facing, or regulated workloads.

Recommended alerts
  • Alert on any VEN tampering event.

  • Raise severity when the affected workload is in selective or full enforcement.

  • Raise severity when tampering is followed by VEN offline, suspension, deactivation, or unpairing events.

Threshold guidance
  • Alert on one event for all production workloads.

  • Alert on one event for any workload in selective or full enforcement.

Investigation questions
  • What user or process performed the action on the host?

  • Are there matching EDR, OS audit, or privileged access events?

  • Did the VEN go offline, suspend, or stop enforcing policy afterward?

  • Was the activity part of approved maintenance or software deployment?

VEN offline events

Recommended Splunk search

index=* sourcetype="illumio:pce" event_type="agent.goodbye" 
What to look for

VEN offline events indicate that the PCE has lost contact with the VEN or that the VEN is no longer reporting as expected.

How this helps spot anomalous activity

A VEN going offline may be caused by normal host shutdowns, reboots, network issues, certificate or connectivity problems, or planned maintenance. However, unexpected offline events can also indicate host compromise, network isolation, service failure, or attempts to disrupt visibility and control.

Recommended alerts
  • Alert when a critical production VEN goes offline.

  • Alert when many VENs go offline in the same environment, application, location, or label group.

  • Alert when offline events follow tampering, suspension, or unpairing activity.

  • Alert when a VEN remains offline beyond the expected maintenance or reboot window.

Threshold guidance
  • Critical workloads: alert immediately or after a short grace period appropriate to the environment.

  • Non-critical workloads: alert when offline status persists beyond normal reboot or maintenance expectations.

  • Group-level alerting: alert when more than a baseline percentage of VENs in an application, environment, or location go offline within 15 minutes.

Investigation questions
  • Is the host powered on and reachable?

  • Was there a network, DNS, proxy, firewall, or certificate change?

  • Did the event coincide with patching, rebooting, or maintenance?

  • Are other VENs in the same location or application also offline?

PCE failed login attempts

Important

This example only works for On-Prem PCE.

Recommended Splunk search

| tstats summariesonly values(Audit.timestamp) AS Timestamp, 
    values(Audit.src_user) AS src_user, 
    values(Audit.user) AS user, 
    values(Audit.src_ip) AS src_ip 
    FROM datamodel=Illumio.Audit 
    WHERE Audit.pce_fqdn="" 
        Audit.org_id="" 
        (Audit.user="" OR Audit.src_user="") 
        (Audit.category="auditable" OR Audit.category="system_events") 
        Audit.event_type="user." 
    BY _time Audit.timestamp Audit.pce_fqdn Audit.href Audit.event_type Audit.status Audit.severity Audit.notification_type 
| rename Audit.pce_fqdn AS PCE, 
    Audit.event_type AS event_type, 
    Audit.notification_type AS notification_type, 
    Audit.severity AS severity, 
    Audit.status AS status, 
    Audit.href AS event_href 
| fillnull value="-" 
| search (event_type="user.") (event_type!="none") (status=failure) (severity=) (notification_type=) 
| table event_href, 
    Timestamp, 
    PCE, 
    event_type, 
    src_user, 
    user, 
    src_ip, 
    notification_type, 
    severity, 
    status 
| rename event_type AS "Event Type", 
    src_user AS "Initiating User", 
    user AS "Target User", 
    src_ip AS "Source IP", 
    notification_type AS "Notification Type", 
    severity AS "Severity", 
    status AS "Status" 
| eval Timestamp = strptime('Timestamp', "%FT%T.%3N%Z") 
| fieldformat "Timestamp" = strftime('Timestamp', "%+") 
| sort -Timestamp 
What to look for

Failed login attempts to the PCE can indicate forgotten passwords, user error, service-account issues, misconfigured identity-provider integration, brute-force attempts, or credential stuffing.

How this helps spot anomalous activity

Repeated failed logins by the same user or from the same source IP may indicate attempted unauthorized access. Failed logins followed by a successful login should be investigated with higher priority.

Recommended alerts
  • Alert when one user has more than five failed login attempts in 10 minutes.

  • Alert when one source IP has failed login attempts against multiple users.

  • Alert when failed logins are followed by a successful login for the same user or source.

  • Alert when administrative accounts have repeated failed logins.

Threshold guidance
  • Standard users: more than five failures in 10 minutes.

  • Administrative users: more than three failures in 10 minutes.

  • Source IP targeting multiple users: more than five users in 10 minutes.

Investigation questions
  • Is the source IP expected for the user or identity provider?

  • Did the failures precede a successful login?

  • Is the account privileged?

  • Are there corresponding identity-provider or VPN authentication events?

PCE Intrusion attempts: core OS events

Note

For SSH logs, configure syslog to send Splunk alerts.

What to look for

Monitor core operating system security events on PCE nodes, depending on deployment model and access to OS-level telemetry. Relevant events may include suspicious login attempts, privilege escalation attempts, unexpected process execution, unauthorized file changes, suspicious network connections, or host-based intrusion detection alerts.

How this helps spot anomalous activity

PCE nodes are critical management infrastructure. Suspicious activity at the OS level may indicate attempted compromise of the management plane. Treat these events with elevated priority and correlate them with PCE application audit logs.

Recommended alerts
  • Alert on failed SSH attempts above baseline.

  • Alert on successful login by an unexpected user or from an unexpected source.

  • Alert on privilege escalation activity outside maintenance windows.

  • Alert on unexpected changes to critical configuration files.

  • Alert on host-based intrusion detection or EDR alerts involving PCE nodes.

Threshold guidance
  • Unexpected successful OS login to a PCE node: alert on one event.

  • Failed SSH attempts: alert when failures exceed baseline, such as more than five from one source in 10 minutes.

  • Privilege escalation activity: alert on one event outside approved maintenance windows.

Investigation questions
  • Is OS-level access to the PCE expected from this user and source?

  • Was there an approved maintenance window?

  • Are there corresponding PCE administrative actions?

  • Are other PCE nodes showing similar events?

  • Should the event be escalated to the security operations or incident response team?

Alert Configuration Best Practices in Splunk

Use these best practices when configuring Splunk alerts for detecting anomalous activity.

Use scheduled searches for operational events

For events such as unpairing, suspension, ignored interfaces, duplicate VENs, and failed logins, scheduled searches are sufficient. Run them every 5-15 minutes depending on your monitoring requirements.

Use real-time or near-real-time alerts for high-risk events

High-risk events such as VEN tampering, enforced VEN unpairing, or suspicious PCE OS-level activity may warrant near-real-time alerting.

Add context to alerts

Alerts should include enough context for triage. Recommended alert fields include:

  • Timestamp

  • PCE FQDN

  • Organization ID

  • Workload hostname

  • Workload href

  • Enforcement mode

  • Event type

  • Acting user

  • Source IP, if available

  • Destination IP and port, for traffic events

  • Workload labels, if available

  • Change ticket reference, if integrated

Suppress known-good noise

Suppress or lower severity for known maintenance windows, approved vulnerability scanners, monitoring tools, deployment automation, and expected provisioning workflows.

Correlate related events

High-confidence alerts often come from sequences of related activity. For example:

  • Failed PCE logins followed by successful login and administrative changes.

  • VEN tampering followed by VEN offline.

  • Ignored interface added followed by reduced traffic visibility.

  • Blocked connections followed by endpoint alerts on the source workload.

  • Port scan behavior followed by attempted access to administrative services.

Align alert routing to ownership

Not every event should go to the same team. Suggested routing:

  • Traffic anomalies: SOC, network security, application owner.

  • VEN operational issues: Illumio operations team, server/platform owner.

  • PCE failed logins: SOC, IAM team, Illumio administrators.

  • PCE OS intrusion attempts: SOC, incident response, platform operations.

Document response actions

Each alert should have a response playbook that includes:

  • Initial triage steps.

  • How to validate whether the event was expected.

  • Who owns the affected workload or platform.

  • Escalation criteria.

  • Containment considerations.

  • Recovery or rollback steps.

Recommended Workflow for Detecting Potentially Anomalous Activity in Splunk

Use this workflow when detecting anomalous activity in Splunk.

Detection and alert workflow
  1. Confirm that Illumio PCE events, VEN operational events, and traffic telemetry are being ingested into Splunk.

  2. Validate field extraction for key fields such as event_type, workload hostname, source IP, destination IP, destination port, user, enforcement mode, labels, PCE FQDN, and org ID.

  3. Start with detection-only dashboards before enabling notifications.

  4. Configure high-confidence alerts first, such as VEN tampering, enforced VEN unpairing, and suspicious PCE access activity.

  5. Baseline activity for at least 2-4 weeks.

  6. Add traffic anomaly alerts with thresholds tuned by application, environment, and workload criticality.

  7. Review alert volume weekly during the tuning period.

  8. Create suppression rules for approved scanners, monitoring tools, maintenance windows, and known automation.

  9. Review alert efficacy with SOC, Illumio administrators, application owners, and platform teams.

  10. Periodically reassess thresholds as the environment, policy model, and workload inventory change.

Summary

Illumio telemetry can help you detect activity that may indicate reconnaissance, attempted lateral movement, unauthorized administrative changes, or reduced segmentation coverage. The most valuable SIEM detections typically combine Illumio traffic data, PCE audit events, VEN operational events, identity logs, endpoint telemetry, and change-management context.

High-priority detections should include:

  • Enforced VEN unpairing or deactivation.

  • VEN tampering.

  • Unexpected VEN suspension.

  • Ignored interfaces on critical workloads.

  • Duplicate VENs involving production systems.

  • VEN offline events affecting critical applications.

  • Repeated PCE failed login attempts.

  • Suspicious OS-level activity on PCE nodes.

  • Blocked traffic patterns consistent with port scans, or unauthorized lateral movement attempts.

Treat the searches and thresholds as starting points. Effective anomaly detection depends on baselining normal behavior, tuning by workload criticality, integrating change context, and reviewing alert quality regularly.