Skip to main content

Streaming your Pega logs

Suggest edit Updated on March 9, 2022

Log streaming gives you continual access to Pega Platform logs in any of your Pega Cloud environments. Integrating an existing log streaming service or AWS S3 bucket with Pega Cloud, enables you to configure dynamic security, application, and computing resource monitoring tools that meet your enterprise requirements.

Before you begin: Integrating third-party services for log streaming requires appropriate access to My Support Portal. To log into the portal and request log streaming access, ensure that you first complete the following tasks:
  • Allow cookies in your browser settings.
  • Contact your Pega Cloud administrator for the cloud-admin (cloud-system contact) role access privileges. For more information, see Support user roles.
Each type of log streaming service has unique access requirements as described in the individual logging service details.

Stream Pega logs to an external Amazon S3 bucket

You can configure your Pega Cloud environment to stream log files to an Amazon Web Services (AWS) S3 bucket in your enterprise AWS account. Streaming logs to your AWS S3 bucket gives you immediate access to your log files without relying on third-party integrations or Pega-provided services.

To complete your log streaming integration with your AWS S3 bucket, make a request by selecting New request in My Support Portal. Include your AWS account information in the request as described below. For the latest documentation on making requests, see Requesting support services.

Pega Cloud supports log streaming integration with your AWS S3 bucket to your Pega Cloud environment using the existing connectivity method already provisioned for your environment.

Caution: Pega supports streaming log files from a single environment to a single AWS S3 bucket or streaming the log files of multiple environments to a single S3 bucket. Streaming logs from multiple environments can lead to security vulnerabilities and resource consumption issues.

Enterprise roles required for this task

This task requires the network security administrator role in your enterprise with access to the AWS policies of an S3 bucket and customer managed keys.

Pega responsibilities

  • SRT sends you two Amazon Resource Names (ARNs) that define the Identity and Access Management (IAM) policies for streaming logs to your S3 bucket.

Client responsibilities

  • You determine the encryption format in which the service delivers logs to your repository. Choose from the following formats:
    • GZIP
    • HADOOP_SNAPPY
    • Snappy
    • ZIP
    • Uncompressed
  • You provide to SRT the ARNs of the following artifacts from your AWS account:
    • Your Amazon S3 custom master keys (CMKs) ARN

      For example, arn:aws:kms:us-west-2:111122223333:key/1234abcd-12ab-34cd-56ef-1234567890ab

      For more information, see Finding the key ID and ARN.

    • Your Amazon S3 bucket name ARN

      For example, arn:aws:s3::::bucket-name

      For more information, see Amazon Resource Names.

  • You note the name of your virtual space environment from which you want to stream your Pega logs.

Streaming your Pega logs to an S3 bucket

To stream your Pega logs directly to your S3 bucket, perform the following task:

  1. Add the following Amazon S3 bucket details to a text file:
    • Amazon S3 bucket name
    • CMK
    • Encryption format for log files sent to your S3 bucket
    • The name of the environment from which you want to stream your logs to an S3 bucket
  2. Log in to your My Support Portal account.
  3. In the header of My Support Portal, click New requestFor something I need.

    Requesting something you need from My Support Portal
    Using My Support Portal to make a request

  4. Use one of the following ways to send the information file securely to SRT:

    • Archiving your Amazon S3 log streaming information with a password:
      1. In the Details section of the request, click Add attachments, and then add a compressed password-protected text file that contains the bucket name, the CMK, the encryption format for your log files, and the environment from which you want to stream your logs.
      2. Continue through the form, and then click Finish to send the archive file with your service request.
      3. Contact the Pega Support team and tell them the password.
    • Allowing Pegasystems to download the file from your personal SFTP server
      1. Upload a text file that contains the bucket name, the CMK, the encryption format for your log files, and the environment from which you want to stream your logs to your personal Secure File Transfer Protocol (SFTP) server.

        For more information about SFTP, see Pega Cloud SFTP service.

      2. Contact the Pega Support team and give them the credentials for the SFTP server.

    After the Pega Cloud team receives your request and your Amazon S3 bucket details, in the request reply, Pega Cloud sends you two Amazon Resource Names (ARNs) that define the IAM policies that you need to stream logs to your Amazon S3 bucket in the following formats:

    <client>-delivery-stream-role ARN
    Grants the streaming service access to your Amazon S3 bucket
    PEGA_CFN_ROLE_ARN
    Declares the resource for the log streaming service
  5. Sign into your Amazon S3 console.
  6. Select the bucket to which you want to add the Amazon S3 log streaming service.
  7. Click Permissions, and then enter the <client>-delivery-stream-role ARN in the bucket policy editor.

    For example,

    {
    "Sid": "PegaKinesisRoleWrite",
    "Effect": "Allow",
    "Principal": {
    "AWS":"<<client>-delivery-stream-role ARN>"
    },
     "Action": [
    "s3:PutObject",
    "s3:PutObjectAcl",
    "s3:ListBucket"
    ],
    "Resource": [
    "arn:aws:s3:::<clientS3bucket>/logs/*",
    "arn:aws:s3:::<clientS3bucket>/logs-processing-failures/*",
    "arn:aws:s3:::<clientS3bucket>"
    ]
    }
    Streaming logs from multiple environments to a single S3 bucket

    If you stream logs from multiple environments, your Resource class must reflect each environment name from which you stream your logs.

    Caution: This configuration is not the default, recommended option for S3 log streaming. Streaming logs from multiple environments to an S3 bucket can cause security vulnerabilities; the log streaming service can access all folders in the bucket: development, testing, and production. You might also reach the AWS resource consumption cap for your S3 bucket. Stream multiple environment logs to a single S3 bucket at your own risk.

    For example,

    {
    "Sid": "PegaKinesisRoleWrite",
    "Effect": "Allow",
    "Principal": {
    "AWS":"<<client>-delivery-stream-role ARN>"
    },
    "Action": [
    "s3:PutObject",
    "s3:PutObjectAcl",
    "s3:ListBucket"
    ],
    "Resource": [
    "arn:aws:s3:::<clientS3bucket>/dev-test/*",
    "arn:aws:s3:::<clientS3bucket>/dev-test-processing-failures/*",
    "arn:aws:s3:::<clientS3bucket>/prod/*",
    "arn:aws:s3:::<clientS3bucket>/prod-processing-failures/*",
    "arn:aws:s3:::<clientS3bucket>/dt1/*",
    "arn:aws:s3:::<clientS3bucket>/dt1-processing-failures/*",
    "arn:aws:s3:::<clientS3bucket>/dt2/*",
    "arn:aws:s3:::<clientS3bucket>/dt2-processing-failures/*",
    "arn:aws:s3:::<clientS3bucket>"
    ]
    }
  8. Click Save changes.

    For more information about adding a policy to your Amazon S3 bucket, see Adding a bucket policy using the Amazon S3 console.

  9. Log in into your AWS KMS console.
  10. In the navigation pane, click Customer managed keys.
  11. Select the S3 CMK.
  12. Select the Key policy tab, and in the key policy editor, add the PEGA_CFN_ROLE_ARN and <client>-delivery-stream-role ARNs.

    For example,

    {
    "Sid": "Enable Initial Create Grant",
    "Effect": "Allow",
    "Principal": {"AWS": 
    "<PEGA_CFN_ROLE_ARN>"},
    "Action": 
    "kms:CreateGrant",
    "Resource": "CMK-Key"},
    {
    "Sid": "Enable Firehose KMS Access",
    "Effect": "Allow",
    "Principal": 
    {
    "AWS": "<<client>-delivery-stream-role ARN>"
    },
    "Action": [
    "kms:Encrypt",
    "kms:Decrypt",
    "kms:ReEncrypt*",
    "kms:GenerateDataKey*",
    "kms:DescribeKey",
    "kms:CreateGrant"
    ],
    "Resource": "CMK-Key"
    }
  13. Click Save changes.

    Your logs begin streaming, and you can now search for your Pega logs in your Amazon S3 bucket. For example, PegaCLUSTER, PegaRULES-ALERTSECURITY, PegaRULES-ALERT and PegaRULESV1.

Stream Pega logs to Splunk

Pega Cloud offers add-on Pega Platform log streaming. By integrating an existing Splunk service with Pega Cloud, you have continuous access to the logs in your Pega Cloud environments. The log streaming service allows to you to efficiently manage your Pega Platform logs dynamically and not have to download logs manually.

To complete a Splunk integration, make a request by selecting New request in My Support Portal. Include your Splunk authentication and connection information in the request. For the latest documentation on making requests, see Requesting support services.

Pega Cloud supports Splunk streaming integration to your Pega Cloud environment using the existing connectivity method already provisioned for your environment.

Requirements and limitations

Streaming Pega Platform logs to a Splunk service requires that an administrator for your Pega Cloud account to complete a cloud change request that includes your Splunk authentication details with which Pega Cloud configures your Splunk connection.

To obtain these details, enable the HTTP Event Collector (HEC) for your organization’s Splunk account. Enabling HEC requires a Splunk administrator role.

Provide the following details to Pega Support:

  • SPLUNK_HEC_URL: The URL address for your Splunk HEC endpoint. Include input- before the URL. For example:

    <input-splunkdomain:port>/services/collector

  • SPLUNK_HEC_TOKEN: The authentication token to permit Pega Cloud access to Splunk for log streaming.
Gathering the required Splunk authentication information to include in your Pega Cloud Service Request

When creating your Splunk HEC token, perform the following tasks your Splunk account:

  • Enable Secure Socket Layer (SSL) during the token creation.
  • Disable the Indexer Acknowledgment.
  • Optional: Edit the SPLUNK_HEC_URL port number during the token creation.
  • Copy the SPLUNK_HEC_TOKEN into a text file.
  • Copy the SPLUNK_HEC_URL into the same text file.

    Include input- before the URL.

    The procedure for enabling HEC for your Splunk account varies by the version of Splunk that you are using. See the Splunk documentation for more information.

Validating your authenticated Splunk connections

Before you send your connection details to permit Pega Cloud access to Splunk, you must validate your Splunk connectivity authentication from your machine to confirm they work. Pega Cloud recommends a temporary SSL test connection to Splunk:

  1. From the command prompt, enter:

    curl -k <SPLUNK_HEC_URL> -H "Authorization: Splunk <SPLUNK_HEC_TOKEN>" -d '{“event": “Pega Splunk Test“}' -v

  2. Confirm the success or failure of the Splunk connection. If you successfully connect, the command returns the following JSON string:

    {"text":"Success","code":0}

    If you are not successful, you must troubleshoot your connection failure with your Splunk account. You must obtain a valid SPLUNK_HEC_URL and SPLUNK_HEC_TOKEN combination before you send these details to Pega Cloud.

Requesting the log streaming service

After you validate your Splunk authentication information, make a request that includes a securely encrypted archive of the Splunk authentication information that Pega Cloud will use to configure your Splunk connection. Use one of the following methods to send the information file securely to SRT:

  • Share a password-protected archive:
    1. Log in to your My Support Portal account.
    2. Select New request in My Support Portal.
    3. Add the text file that contains the SPLUNK_HEC_TOKEN and SPLUNK_HEC_URL to a compressed archive that is password-protected.
    4. Send the archive file with your service request.
    5. Contact the Pega Support team by email or call and tell them the password.
  • Allow Pegasystems Inc. to download the file from your personal Secure File Transfer Protocol (SFTP) server
    1. Log in to your My Support Portal account.
    2. Select New request in My Support Portal.
    3. Upload the text file that contains the SPLUNK_HEC_TOKEN and SPLUNK_HEC_URL to your personal Pega Cloud SFTP server. For more information about SFTP, see Pega Cloud SFTP service.
    4. Contact the Pega Support team by email or by calling and give them the credentials for the SFTP server.
Result: After the Pega Cloud team receives your request and the authentication details file, Pega Cloud authenticates Splunk connectivity from your Pega Cloud environment.

After authenticating connectivity, the Pega Cloud team completes the add-on integration with Splunk and notifies you that your environment has been updated.

Confirming that the log streaming service is active

After you receive confirmation from the Pega Cloud team that the Splunk service integration is complete, your Pega Platform logs are searchable in the Splunk GUI. For example, PegaCLUSTER and PegaRULESV1.

Did you find this content helpful? YesNo

Have a question? Get answers now.

Visit the Support Center to ask questions, engage in discussions, share ideas, and help others.

We'd prefer it if you saw us at our best.

Pega.com is not optimized for Internet Explorer. For the optimal experience, please use:

Close Deprecation Notice
Contact us