Skip to main content

Streaming Pega logs to an external Amazon S3 bucket

Suggest edit Updated on June 2, 2022

You can configure your Pega Cloud environment to stream log files to an Amazon Web Services (AWS) S3 bucket in your enterprise AWS account. Streaming logs to your AWS S3 bucket gives you immediate access to your log files without relying on third-party integrations or Pega-provided services.

Integrating third-party services for log streaming requires appropriate access to My Support Portal. To log into the portal and request log streaming access, ensure that you first complete the following tasks:

  • Allow cookies in your browser settings.
  • Contact your Pega Cloud administrator for the cloud-admin (cloud-system contact) role access privileges. For more information, see Support user roles.

To complete your log streaming integration with your AWS S3 bucket, make a request by selecting New request in My Support Portal. Include your AWS account information in the request as described below. For the latest documentation on making requests, see Requesting support services.

Pega Cloud supports log streaming integration with your AWS S3 bucket to your Pega Cloud environment using the existing connectivity method already provisioned for your environment.

Caution: Pega supports streaming log files from a single environment to a single AWS S3 bucket or streaming the log files of multiple environments to a single S3 bucket. Streaming logs from multiple environments can lead to security vulnerabilities and resource consumption issues.

Enterprise roles required for this task

This task requires the network security administrator role in your enterprise with access to the AWS policies of an S3 bucket and customer managed keys.

Pega responsibilities

  • SRT sends you two Amazon Resource Names (ARNs) that define the Identity and Access Management (IAM) policies for streaming logs to your S3 bucket.

Client responsibilities

  • You determine the encryption format in which the service delivers logs to your repository. Choose from the following formats:
    • GZIP
    • HADOOP_SNAPPY
    • Snappy
    • ZIP
    • Uncompressed
  • You provide to SRT the ARNs of the following artifacts from your AWS account:
    • Your Amazon S3 custom master keys (CMKs) ARN

      For example, arn:aws:kms:us-west-2:111122223333:key/1234abcd-12ab-34cd-56ef-1234567890ab

      For more information, see Finding the key ID and ARN.

    • Your Amazon S3 bucket name ARN

      For example, arn:aws:s3::::bucket-name

      For more information, see Amazon Resource Names.

  • You note the name of your virtual space environment from which you want to stream your Pega logs.

Streaming your Pega logs to an S3 bucket

To stream your Pega logs directly to your S3 bucket, perform the following task:

  1. Add the following Amazon S3 bucket details to a text file:
    • Amazon S3 bucket name
    • CMK
    • Encryption format for log files sent to your S3 bucket
    • The name of the environment from which you want to stream your logs to an S3 bucket
  2. Log in to your My Support Portal account.
  3. In the header of My Support Portal, click New requestFor something I need.

    Requesting something you need from My Support Portal
    Using My Support Portal to make a request

  4. Use one of the following ways to send the information file securely to SRT:

    • Archiving your Amazon S3 log streaming information with a password:
      1. In the Details section of the request, click Add attachments, and then add a compressed password-protected text file that contains the bucket name, the CMK, the encryption format for your log files, and the environment from which you want to stream your logs.
      2. Continue through the form, and then click Finish to send the archive file with your service request.
      3. Contact the Pega Support team and tell them the password.
    • Allowing Pegasystems to download the file from your personal SFTP server
      1. Upload a text file that contains the bucket name, the CMK, the encryption format for your log files, and the environment from which you want to stream your logs to your personal Secure File Transfer Protocol (SFTP) server.

        For more information about SFTP, see Pega Cloud SFTP service.

      2. Contact the Pega Support team and give them the credentials for the SFTP server.

    After the Pega Cloud team receives your request and your Amazon S3 bucket details, in the request reply, Pega Cloud sends you two Amazon Resource Names (ARNs) that define the IAM policies that you need to stream logs to your Amazon S3 bucket in the following formats:

    <client>-delivery-stream-role ARN
    Grants the streaming service access to your Amazon S3 bucket
    PEGA_CFN_ROLE_ARN
    Declares the resource for the log streaming service
  5. Sign into your Amazon S3 console.
  6. Select the bucket to which you want to add the Amazon S3 log streaming service.
  7. Click Permissions, and then enter the <client>-delivery-stream-role ARN in the bucket policy editor.

    For example,

    {
    "Sid": "PegaKinesisRoleWrite",
    "Effect": "Allow",
    "Principal": {
    "AWS":"<<client>-delivery-stream-role ARN>"
    },
     "Action": [
    "s3:PutObject",
    "s3:PutObjectAcl",
    "s3:ListBucket"
    ],
    "Resource": [
    "arn:aws:s3:::<clientS3bucket>/logs/*",
    "arn:aws:s3:::<clientS3bucket>/logs-processing-failures/*",
    "arn:aws:s3:::<clientS3bucket>"
    ]
    }
    Streaming logs from multiple environments to a single S3 bucket

    If you stream logs from multiple environments, your Resource class must reflect each environment name from which you stream your logs.

    Caution: This configuration is not the default, recommended option for S3 log streaming. Streaming logs from multiple environments to an S3 bucket can cause security vulnerabilities; the log streaming service can access all folders in the bucket: development, testing, and production. You might also reach the AWS resource consumption cap for your S3 bucket. Stream multiple environment logs to a single S3 bucket at your own risk.

    For example,

    {
    "Sid": "PegaKinesisRoleWrite",
    "Effect": "Allow",
    "Principal": {
    "AWS":"<<client>-delivery-stream-role ARN>"
    },
    "Action": [
    "s3:PutObject",
    "s3:PutObjectAcl",
    "s3:ListBucket"
    ],
    "Resource": [
    "arn:aws:s3:::<clientS3bucket>/dev-test/*",
    "arn:aws:s3:::<clientS3bucket>/dev-test-processing-failures/*",
    "arn:aws:s3:::<clientS3bucket>/prod/*",
    "arn:aws:s3:::<clientS3bucket>/prod-processing-failures/*",
    "arn:aws:s3:::<clientS3bucket>/dt1/*",
    "arn:aws:s3:::<clientS3bucket>/dt1-processing-failures/*",
    "arn:aws:s3:::<clientS3bucket>/dt2/*",
    "arn:aws:s3:::<clientS3bucket>/dt2-processing-failures/*",
    "arn:aws:s3:::<clientS3bucket>"
    ]
    }
  8. Click Save changes.

    For more information about adding a policy to your Amazon S3 bucket, see Adding a bucket policy using the Amazon S3 console.

  9. Log in into your AWS KMS console.
  10. In the navigation pane, click Customer managed keys.
  11. Select the S3 CMK.
  12. Select the Key policy tab, and in the key policy editor, add the PEGA_CFN_ROLE_ARN and <client>-delivery-stream-role ARNs.

    For example,

    {
    "Sid": "Enable Initial Create Grant",
    "Effect": "Allow",
    "Principal": {"AWS": 
    "<PEGA_CFN_ROLE_ARN>"},
    "Action": 
    "kms:CreateGrant",
    "Resource": "CMK-Key"},
    {
    "Sid": "Enable Firehose KMS Access",
    "Effect": "Allow",
    "Principal": 
    {
    "AWS": "<<client>-delivery-stream-role ARN>"
    },
    "Action": [
    "kms:Encrypt",
    "kms:Decrypt",
    "kms:ReEncrypt*",
    "kms:GenerateDataKey*",
    "kms:DescribeKey",
    "kms:CreateGrant"
    ],
    "Resource": "CMK-Key"
    }
  13. Click Save changes.

    Your logs begin streaming, and you can now search for your Pega logs in your Amazon S3 bucket. For example, PegaCLUSTER, PegaRULES-ALERTSECURITY, PegaRULES-ALERT and PegaRULESV1.

Did you find this content helpful? YesNo

Have a question? Get answers now.

Visit the Support Center to ask questions, engage in discussions, share ideas, and help others.

We'd prefer it if you saw us at our best.

Pega.com is not optimized for Internet Explorer. For the optimal experience, please use:

Close Deprecation Notice
Contact us