Skip to main content

Security Advisory: Apache Log4j JNDI Zero Day Vulnerability

Suggest edit Updated on January 13, 2022

+

This content applies to On-Premises Services and Cloud environments.

Important: This is the primary authoritative source for information related to the status of Pega's products and services.Important: Pega is closely following this volatile situation.  We continue to patch our products and services as new vulnerabilities in Log4j are identified, and will continue to post updates in this Advisory as they occur.  Log4j 2.17.1 was just released.  Pega is actively investigating the impact of this version, and will be providing hotfixes (click here for details).

Overview

A zero-day vulnerability was identified in the Apache Log4j logging software on Friday, Dec. 10 (CVE-2021-44228)A related Log4j vulnerability was identified on Tuesday, Dec. 14 (CVE-2021-45046), and a third was identified on Friday, December 17 (CVE-2021-45105).  These vulnerabilities could allow malicious actors to take control of organizational networks using Log4j.  The Log4j software is ubiquitously used by most organizations around the world.  

Pega software can use the Log4j component in two places:  the Pega Platform software and Pega's Stream service.

  • Log4j is embedded within our Pega Platform product to allow clients to track and record platform activity. 
  • The Pega Stream service enables the asynchronous flow of data between processes in the Pega Platform. The Stream service is a multi-node component that is based on Apache Kafka.

Click here to scroll down to information about:

Pega Cloud Clients

Pega On-Premises or Client-managed Cloud clients

Additional Services:  WFI, Pega Chat, Co-Browse, Digital Messaging, Voice AI, and others

Testing

Hotfix Details

 

How Pega is mitigating these vulnerabilities for Pega Cloud clients

To mitigate these vulnerabilities, Pega has applied the following defense-in-depth approach to our Pega Cloud clients.

For Pega Platform

  • We applied security controls at the network level using AWS Web Application Firewall (WAF) on the afternoon of Friday, Dec. 10. 
  • We have disabled the vulnerable portion of Apache Log4j (JNDI) from Pega Platform, followed by a rolling restart, for Pega Cloud clients as of Sunday, December 12, 2021, at 6 pm ET. Through initial testing, we believe this action will not have an adverse impact on the normal use of Pega Platform. 
  • We have scanned all Pega Cloud sites with the Tenable.io Web Application Scanner for Log4j vulnerabilities, and verified that mitigations are in place.
  • The Hazelcast functionality is being updated to mitigate the Log4j vulnerabilities.

For the Stream service, using Apache Kafka:

  • Most Pega Cloud environments, beginning with Pega Platform version 8.4.0, use AWS-managed Apache Kafka, which does not contain the vulnerable log4j libraries.
  • The Kafka service is not directly accessible from the Internet, and is only available as an internal service to the Pega Platform environments on Pega Cloud.
  • As an additional failsafe, we have disabled the vulnerable JNDI portion of Apache Log4j from the Apache Kafka distribution for Pega Cloud clients.

Pega Cloud Web Application Firewall (WAF) rules

Pega Cloud uses the AWS WAF rules.  AWS does not publish the details of these rules, but more information is available at the AWS Managed Rules changelog.

Trend Micro Intrusion Prevention System (IPS)

Pega has applied Trend Micro Intrusion Prevention System (IPS) rules at the host-based agent level on Saturday, Dec. 11, as soon as it became available, to further block malicious attempts.   For details, see their article Security Alert:  Apache Log4j “Log4Shell” Remote Code Execution 0-Day Vulnerability.

For Pega on-premises and self-managed cloud customers 

This vulnerability can affect Pega clients running on-premises or self-managed cloud clients using Pega Platform version 7.3.x - 8.6.x.

Note: For Pega Platform versions prior to 7.3.x, see Security Advisory:  Apache Log4j 1.2 JMSAppender vulnerability

Versions 8.3.x - 8.6.x of the Pega Platform include the Apache Kafka distribution that contains the vulnerable Log4j JNDI libraries.

In addition, it is possible that vendor platforms which clients are using with their Pega software (such as WebSphere or WebLogic) are also affected by the Log4J JNDI vulnerability.  Pega strongly recommends that clients check with their vendors for any required mitigations, and work with their IT and security staff to confirm that other (non-Pega) products are not inadvertently introducing vulnerabilities.

For Pega Platform

Notice: Pega hotfixes are available for our Pega Platform (8.x versions) for the following vulnerabilities:

CVE

Fixed in Apache Log4j version:

CVE-2021-44228

2.15

CVE-2021-45046

2.16

CVE-2021-45105

2.17

Clients can request these hotfixes through My Support Portal.  Please see details at  Pega Security Advisory – Apache Log4j 2.17 Vulnerability Hotfixes.

The above Hotfix Advisory supercedes our original Pega Platform Hotfix Advisory, which was based on Apache Log4j 2.15 and 2.16.  All clients should apply the latest Pega Platform hotfix, even if they had applied the earlier Pega Platform hotfix, as the 2.17 hotfix addresses all the abovevulnerabilities.

CVE-2021-45105 describes a vulnerability that could lead to Distributed Denial of Service (DDoS) attacks.  Applications which are not Internet-facing should be at less risk for this vulnerability.  To limit risk from this vulnerability, Pega strongly recommends that clients avoid exposing unnecessary parts of their systems to the Internet, and protect their Internet-facing features by using security functionality such as WAF or IPS.

Note: Pega will be releasing Pega Platform hotfixes based on Apache Log4j 2.17.1, which addresses CVE-2021-44832.  The target date for releasing these hotfixes will be the week of January 17, 2022, in a separate Hotfix Advisory.

This vulnerability can only be exploited if the adversary has already gained access to a client’s system through another means (which would indicate a much larger security issue for the organization).  Therefore, these hotfixes (based on Log4j 2.17.1) will only be available for the latest patch release of each Pega Platform version, and clients must evaluate whether they require this fix.  Clients who are on prior patch versions must upgrade to the latest patch version in order to receive and apply this hotfix. 

Pega clients, both Pega Cloud and on-premise/client-managed cloud, who are on the latest patch version will need to request this hotfix through GCS.  Clients who are not on the latest patch release must upgrade to the most current patch release before requesting the hotfix.

 

Notice: Pega Platform hotfixes for version 7.3, 7.3.1, and 7.4 are available for the above-listed vulnerabilities.  See details at Pega 7 Hotfix Advisory for Apache Log4j Zero Day Vulnerability.    

NOTE:  If clients apply a hotfix for a particular Pega Platform version, and then later update their systems to a newer Platform version, they must apply all critical hotfixes to this newer version.

The below instructions describe how clients can disable the vulnerable JNDI portion of Apache Log4j functionality. We urge all Pega on-premises and self-managed cloud clients to apply the correct version of the latest hotfix to their environments, or take the below manual action immediately on their Pega environments.   In addition, clients should also follow the security recommendations and guidelines from your organization. 

Manual Removal of JNDILookup.class

The following steps remove the JNDILookup.class from the pr_engineclasses table. 

Note: This action will not have an adverse impact on the normal use of Pega Platform unless clients have applied the prior Pega Platform hotfixes based on Log4j 2.16.  Clients who installed those prior hotfixes and then manually deleted the JNDILookup.class can run into an issue where log files may be written to the wrong location.  All clients should apply the Pega Platform hotfixes which are based on Log4j 2.17 (linked above).  The Log4j 2.17 hotfixes will fix this issue, in addition to providing more up-to-date protection from vulnerabilities.

1.  Enter the following SQL statement, to confirm one or more of those classes are present in the DB.  You should see at least one record returned.

select pzjar,pzpackage,pzclass,pzlastmodified,pzmoduleversion,pzcodesetversion,pzpatchdate from <Rules Schema Name>.pr_engineclasses where pzclass = 'JndiLookup.class' and pzpackage = 'org/apache/logging/log4j/core/lookup';

2.  Back up the pr_engineclasses table.

3.  Delete the JNDILookup.class.

delete from <Rules Schema Name>.pr_engineclasses where pzclass = 'JndiLookup.class' and pzpackage = 'org/apache/logging/log4j/core/lookup';

4.  Do a full cluster restart.  NOTE:  a Docker restart is not sufficient.  A rolling restart is fine.

5.  Run the select statement again to confirm that the class is removed.  No results should be returned.

select pzjar,pzpackage,pzclass,pzlastmodified,pzmoduleversion,pzcodesetversion,pzpatchdate from <Rules Schema Name>.pr_engineclasses where pzclass = 'JndiLookup.class' and pzpackage = 'org/apache/logging/log4j/core/lookup';

 

For the Stream service, using Apache Kafka

The Apache Kafka service should not be directly accessible from the Internet, and should only be available as an internal service accessible to the Pega Platform environments via a private network.  Pega strongly recommends ensuring that this measure is in place.

Pega hotfixes are available for our Stream Service (8.x versions) for the following vulnerabilities:

CVE

Fixed in Apache Log4j version:

CVE-2021-44228

2.15

CVE-2021-45046

2.16

CVE-2021-45105

2.17

Important: Clients can request these hotfixes through My Support Portal.  Please see details at  Stream Security Advisory – Apache Log4j 2.17 Vulnerability Hotfixes.Notice: The above Hotfix Advisory supercedes our original Stream Hotfix Advisory, which was based on Apache Log4j 2.15.  All clients should apply the latest Stream service hotfix, even if they had applied the earlier Stream service hotfix, as the 2.17 hotfix addresses all the above vulnerabilities.

We urge all Pega on-premises and self-managed cloud clients to apply the appropriate latest hotfix to their environments, or take the below manual action immediately on their Pega environments.   

If an immediate remediation is required, follow the below procedure on your Pega Platform Stream nodes to replace the vulnerable JNDI libraries with the upgraded files from Apache Kafka. 

The Stream nodes are configured with the -DNodeType=Stream argument.  Important:  You must follow this process for each Stream node. 

  1. Log in to your application server using the command line.
  2. Stop the application server process.
  3. Navigate to java_ee_server_root/kafka-<kafka-version>/libs, where <kafka-version> is either 1.1.0.5 or 1.1.0.4
  4. Delete the following files:
  • log4j-api-2.11.1.jar
  • log4j-core-2.11.1.jar
  • log4j-slf4j-impl-2.11.1.jar
  1. Download the upgraded log4j files from Apache.  These are the files that Apache has provided to mitigate this vulnerability:
  • log4j-api-2.16.0.jar
  • log4j-core-2.16.0.jar
  • log4j-slf4j-impl-2.16.0.jar
  1. Copy these files to java_ee_server_root/kafka-<kafka_version>/libs
  2. Verify that java_ee_server_root/kafka-<kafka_version>/libs contains only the following jars related to log4j:
  • log4j-api-2.16.0.jar
  • log4j-core-2.16.0.jar
  • log4j-slf4j-impl-2.16.0.jar
  • (possibly) kafka-log4j-appender-1.1.0.jar

NOTE:   The 2.16.0 versions of these files are the latest version that Pega has tested for the above manual mitigation steps.  Since Pega provides a hotfix for the 2.17 versions of Log4j, we have not tested the above manual stepsfor that version. 

The verification can be done by running the following command:

 $ ls | grep 'log4j'
 log4j-api-2.16.0.jar
 log4j-core-2.16.0.jar
 log4j-slf4j-impl-2.16.0.jar
 kafka-log4j-appender-1.1.0.jar

  1. Restart the application server.

 

Additional Pega Services

Pega Workforce Intelligence (WFI)

Servers vulnerable to the Log4J JNDI issue (CVE-2021-44228) were disabled as of Saturday, December 11, and have been hotfixed.  Servers vulnerable to the additional Log4J vulnerability (CVE-2021-45046) were disabled as of Wednesday, December 15, and have been hotfixed. 

NOTE:  During this process, data is still being collected on client systems, and will be available when the servers are back online. 

A hotfix to address CVE-2021-45105  (based on Apache Log4j 2.17)  has been applied to all WFI client environments.  In addition, AWS WAF has been deployed across the AWS account for all client environments.

Pega Chat and Pega Co-Browse

Pega has completed our analysis of Pega Chat and Co-Browse, and has confirmed that Log4J is not included in these products. 

Pega Digital Messaging

The Log4j JNDI vulnerabilities in this service are addressed through our Pega Platform hotfixes.

Pega Voice AI

External-facing servers vulnerable to the Log4J JNDI issue were disabled as of Monday, December 13; they are being hotfixed before being re-enabled. 

BIX

The Log4j JNDI vulnerabilities in this service are addressed through our Pega Platform hotfixes.

Document Processing Service

Servers vulnerable to the Log4J JNDI issue (CVE-2021-44228) and the additional Log4J vulnerability (CVE-2021-45046) have been hotfixed to mitigate this risk.

PDC

The Log4j JNDI vulnerabilities in this service are addressed through our Pega Platform hotfixes.

Pega Platform Personal Edition

The ability to download the Pega Platform Personal Edition has been temporarily removed while Pega mitigates the Log4j risks. 

Pega is in the process of publishing an updated version of the Personal Edition which will mitigate the Log4J JNDI issue (CVE-2021-44228) and the additional Log4J vulnerability (CVE-2021-45046).  Pega strongly recommends that any client using Personal Edition shut it down until they can download this mitigated version, and then replace their existing Personal Edition with the mitigated version.

Pega Search Functionality

Pega Search functionality uses a product called Elasticsearch.  Clients most commonly access this functionality one of two ways:

1.  Embedded in the Pega application

Legacy setups which have Elasticsearch embedded in Pega Platform use the Log4j component that is part of Pega Platform.  Therefore, applying the Pega Platform hotfixes will mitigate the Log4j vulnerabilities in this situation.

2.  Connecting to a node running an Elasticsearch Docker image

For clients running a client-managed cloud environment, Pegasystems provides multiple Docker images, including separate Docker images for Pega Platform, Pega Search, and other services.  Pegasystems Search is based on the Elasticsearch product.  We have updated our Pegasystems Search Docker images to incorporate a JVM argument, as recommended by Elasticsearch:

https://discuss.elastic.co/t/apache-log4j2-remote-code-execution-rce-vulnerability-cve-2021-44228-esa-2021-31/291476

To mitigate the risk for Elasticsearch, clients need to install only the updated mitigated Docker image for Search (not the Pega Platform Docker image), available from the Pegasystems Search  page.   

For versions 8.2 – 8.6, Pega has updated both the main major/minor version of Pegasystems Search (example:  “8.5”) and the latest patch version of that major/minor version (example “8.5.5”).  Either of these Search Docker images are applicable to all Pega patch versions within the minor version (example:  both “8.5” and “8.5.5” apply to 8.5, 8.5.1, 8.5.2, 8.5.3, 8.5.4, and 8.5.5). 

Pega strongly recommends that clients running Pega software in a client-managed cloud download the appropriate version of the Search image and install it.

NOTE:   The JVM argument fix is the solution recommended by Elasticsearch.  A scan of the software will still report the older Log4j file, but its vulnerability is mitigated by the recommended fix.

Robotics

Robotic Process Automation (Robotics) is made up of several products. 

  • Robotic Runtime, Robot Studio, and Sync Server do not use Log4j.
  • Robot Manager is a Pega Platform application, so the Log4j JNDI vulnerabilities in this service are addressed through our Pega Platform hotfixes.

PRPCServiceUtils

Pega has disabled the download links for this component on Pega Marketplace, and is in the process of developing and publishing an updated version of the PRPCServiceUtils which will mitigate the Log4J JNDI issue (CVE-2021-44228) and the additional Log4J vulnerability (CVE-2021-45046).  Pega strongly recommends that clients not use PRPCServiceUtils until they can download this mitigated version, and then replace their existing PRPCServiceUtils with the mitigated version.

prpcUtils

prpcUtils is used to install Pega Platform, Pega applications, and (sometimes) hotfixes.  (Hotfix Manager and Designer Studio do not use this utility.)  It is not constantly running, but only used when doing these installations or updates.   To mitigate risk, clients should use other Pega-provided utilities such as Hotfix Manager or Deployment Manager.

Hazelcast

Hazelcast may be vulnerable if it is deployed in client-server mode.  Pega is providing separate hotfixes to address Hazelcast vulnerabilities.  These hotfixes are still in development, and will be provided when available.

Testing for the Log4j vulnerabilities

Pega has scanned all Pega Cloud client sites with the Tenable.io Web Application Scanner for Log4j vulnerabilities, and verified that mitigations are in place. 

To determine that the hotfixes have been installed properly, use the System Scanner.

Clients who have questions about validating that the vulnerability has been addressed by application of the Pega hotfixes should work with their security organization, and have their security teams use their preferred scanning tools to test.

Pega Hotfix Details

The following vulnerabilities have been discovered in Apache Log4j:

CVE

Fixed in Apache Log4j version:

CVE-2021-44228

2.15

CVE-2021-45046

2.16

CVE-2021-45105

2.17

Pega hotfixes for our Pega Platform  Infinity  software (8.x versions) which leverage the Apache Log4j version 2.17 are available.   See details at Pega Security Advisory – Apache Log4j 2.17 Vulnerability Hotfixes.

Hotfixes for the Pega 7 versions of Pega Platform (7.3, 7.3.1, and 7.4) which leverage the Apache Log4j version 2.17 are available.   See details at Pega 7 Hotfix Advisory for Apache Log4j Zero Day Vulnerability.    

Pega hotfixes for our Stream Service (8.x versions) which leverage the Apache Log4j version 2.17 are available.  See details at  Stream Security Advisory – Apache Log4j 2.17 Vulnerability Hotfixes.

Notice:

Clients can submit a hotfix request for any of these types of hotfixes by using My Support Portal.

  

 

 

 

 

Document Revisions

January 13, 2022 – 5:45 pm EST  |  Updated date in note about Pega Platform 2.17.1 hotfixes.

January 10, 2022 – 5:45 pm EST  |  Updated information about Pega Search Functionality.

January 6, 2022 – 5:45 pm EST  |  Added note about Pega Platform 2.17.1 hotfixes.

January 4, 2022 – 7:15 pm EST  |  Added note about not doing the Pega Platform manual steps if you’ve already applied the 2.16 hotfix.

December 29, 2021 – 3:30 pm EST  |  Log4j 2.17 hotfixes are now available for Pega Platform, both the 8.x versions, as well as 7.3, 7.3.1, and 7.4.  Pega is investigating Log4j version 2.17.1. 

December 28, 2021 – 2:00 pm EST  |  The Log4j 2.17 hotfix is being applied to WFI client environments.

December 27, 2021 - 3:00 pm EST  |   The Pega Platform hotfix for 7.3.1 is now available.   Added information about Pegasystems Search Docker Images and prpcUtils.

December 23, 2021 – 2:00 pm EST  |  Added note about CVE-2021-45105 being a DDOS vulnerability, and that clients should avoid unnecessarily exposing their systems to the Internet.

December 22, 2021 – 5:30 pm EST  |  Added information about Hazelcast, and a note about Pega Platform 7.3.x and 7.4.x hotfixes.

December 21, 2021 – 5:45pm EST  |  Added information about Pega Cloud clients being scanned with Tenable.io, and about WAF rules and Trend Micro rules.  Links are provided to the new hotfixes are available for the Stream Service; all clients should apply those.  Testing information has been added.

December 20, 2021 – 5:15 pm EST  | Added link to Apache Log4j JMSAppender advisory, and a statement that we are creating hotfixes based on Log4j 2.17 for both Pega Platform and Stream service, and hotfixing WFI.  Added information about Robotics and PRPCServiceUtils.

December 17, 2021 – 3:30 pm EST  | Added link to Stream Services (Kafka) hotfixes.  Also updated information about PDC, Personal Edition, and Docker images.  Added note about clients who update their systems to a later patch must also apply the hotfix that matches that later version.

December 16, 2021 – 5:45 pm EST  |  Added link to Pega Platform hotfixes.  Also updated information about WFI and Chat/Co-Browse; added information about BIX and Document Processing Service.

December 15, 2021 – 12:30 pm EST | Added information about CVE-2021-45046, and updated the Kafka section to reflect that clients should download version 2.16 of the Apache files

December 14, 2021 – 6:24 pm EST  |  Included information about Additional Pega Services, and added links in the Overview for easier scrolling

December 13, 2021 – 7:55 pm EST  |  Added information about Kafka remediation and Hotfixes.

December 11, 2021 – 5:23 pm EST  |  Published article.

 

 

Did you find this content helpful? YesNo

89% found this useful

Have a question? Get answers now.

Visit the Collaboration Center to ask questions, engage in discussions, share ideas, and help others.

Ready to crush complexity?

Experience the benefits of Pega Community when you log in.

We'd prefer it if you saw us at our best.

Pega.com is not optimized for Internet Explorer. For the optimal experience, please use:

Close Deprecation Notice
Contact us