Skip to main content

Integrations

Collect Data from the Amazon S3 Bucket

A log source with the “Illumio ASP V2” log source type is required to collect data from the Amazon S3 bucket.

If a log source with “Illumio ASP V2” is not available, create it by following the steps listed in Add the PCE as a Log Source in QRadar.

You can provide any valid log source identifier for the “Illumio ASP V2” log source type if you are using it only to collect data from the Amazon S3 bucket.

You can use the following ways to enable QRadar to receive events from the Amazon S3 buckets:

  • With an SQS queue

  • With a directory prefix

Collect Data from the Amazon S3 Bucket with an SQS Queue

Use the following steps to create a log source for collecting Illumio events from Amazon S3.

  1. On the Admin tab in QRadar, select Log Sources > Add, and enter the following values:

    1. For Log Source type, select Amazon AWS CloudTrail.

    2. For Protocol type, select Amazon AWS S3 REST API.

    3. Add a name.

    4. Add a description.

    5. Set Enabled to True.

    6. Set Coalescing Events to False.

    7. Set Store Event Payloads to True.

    8. For Log Source Identifier, enter the same value as you did for the name, to avoid confusion.

  2. Continue adding the following values:

    1. For Authentication Method, select Access Key ID/Secret Key.

    2. For Access Key ID, select AWS S3 bucket access key ID.

    3. For Secret Key, select AWS S3 bucket Secret Key.

    4. For S3 Collection Method, select SQS Event Notifications.

    5. For SQS Queue URL, enter the URL of the created SQS Queue.

    6. For Region Name, enter the AWS Region of the SQS Queue resource.

    7. For Bucket Name, enter the S3 bucket name.

    8. For Event Format, select LINEBYLINE.

    9. For User as a Gateway Log Source, select True.

  3. For Log Source Identifier Pattern, enter (=.*) after the Illumio log source identifier, such as {ILLUMIO_LOG_SOURCE_IDENTIFIER}=.* You can find the log source identifier value from the “Illumio ASP v2” log source. For example, if Illumio’s log source identifier is core0-2x2devtest59, then enter core0-2x2devtest59=. * in this field.

    Note

    The Gateway log source collects events from the Amazon S3 bucket and those events can be parsed as "Illumio ASP V2" log source type events because the Illumio ASP V2 log source type's identifier is used while configuring Gateway Log Source.

  4. Set Show Advanced Options to True.

    1. File Pattern: .*\.gz (To consume only .gz files from the S3 bucket)

    2. File Pattern: .* (To consume all files from the S3 bucket)

  5. Set Automatically Acquire Server Certificate(s) to Yes.

  6. Set a value for Recurrence. This designates how often the Amazon AWS S3 REST API Protocol connects to the Amazon cloud API, checks for new files, and if they exist, retrieves them. Every access to an AWS S3 bucket incurs a cost to the account that owns the bucket. The time interval can include values in hours (H), minutes (M), or days (D). For example: 2H = 2 hours, 15M = 15 minutes, 30 = 30 seconds.

  7. Set the value for EPS Throttle. This is the maximum number of events per second (EPS) that this log source should not exceed. (The default value is 5000.)

  8. In the Admin tab, click Deploy Changes.

Collect Data from the Amazon S3 Bucket with a Directory Prefix

Use the following steps to create a log source for collecting Illumio events from Amazon S3.

  1. On the Admin tab in QRadar, select Log Sources > Add and enter the following:

    1. For Log Source type, select Amazon AWS CloudTrail.

    2. For Protocol type, select Amazon AWS S3 REST API.

    3. Add a name.

    4. Add a description.

    5. Set Enabled to True.

    6. Set Coalescing Events to False.

    7. Set Store Event Payloads to True.

  2. Continue entering values:

    1. For Log Source Identifier, enter the same value as you did for the name, to avoid confusion.

    2. For Authentication Method, select Access Key ID/Secret Key.

    3. For Access Key ID, select AWS S3 bucket access key ID.

    4. For Secret Key, select AWS S3 bucket Secret Key.

    5. For S3 Collection Method, use a specific prefix - Single Account/Region Only.

    6. For Bucket Name, enter the S3 bucket name.

    7. For Directory Prefix, enter the root directory location on the AWS S3 bucket from which the files are retrieved. (Directories are separated by '/'.)

    8. For Signature Version, select AWS Signature V2.

    9. For Event Format, select LINEBYLINE.

    10. For User as a Gateway Log Source, select True.

  3. For Log Source Identifier Pattern, enter (=.*) after Illumio log source identifier, such as {ILLUMIO_LOG_SOURCE_IDENTIFIER}=.* You can find the log source identifier value from the “Illumio ASP v2” log source. For example, if Illumio’s log source identifier is core0-2x2devtest59, then enter core0-2x2devtest59=. * in this field.

    Note

    The Gateway log source collects events from the Amazon S3 bucket and those events can be parsed as "Illumio ASP V2" log source type events because the Illumio ASP V2 log source type's identifier is used while configuring Gateway Log Source.

  4. Set Show Advanced Options to True.

    1. File Pattern: .*\.gz (To consume only .gz files from the S3 bucket)

    2. File Pattern: .* (To consume all files from the S3 bucket)

  5. Set Automatically Acquire Server Certificate(s) to Yes.

  6. Set the value for Recurrence. This designates how often the Amazon AWS S3 REST API Protocol connects to the Amazon cloud API, checks for new files, and if they exist, retrieves them. Every access to an AWS S3 bucket incurs a cost to the account that owns the bucket. The time interval can include values in hours (H), minutes (M), or days (D). For example: 2H = 2 hours, 15M = 15 minutes, 30 = 30 seconds.

  7. Set a the value for EPS Throttle. This designates the maximum number of events per second (EPS) that this log source should not exceed. (The default value is 5000.)

  8. In the Admin tab, click Deploy Changes.

Add S3 Bucket Certificates

After you create a log source, make sure that the SSL certificates of the S3 buckets are present in QRadar. If the certificates are not present, the data from the S3 bucket will not be collected.

Use the following procedure to add certificates to the S3 bucket:

  1. Log into QRadar using a secure connection.

  2. Run the following command:

    /opt/qradar/bin/getcert.sh <bucket name>.s3.amazonaws.com