Skip to main content

Externally managed Stream service

Suggest edit Updated on February 28, 2020

You can now configure your Stream service to connect to an external Kafka cluster. Externally managed Stream service clusters operate independently of your Pega Platform™ application. This means that such tasks as an upgrade or hotfix deployment require restating only the nodes that manage the application, without any impact on the Stream service operations.

Connecting the Stream service to an externally managed Kafka cluster

For on-premise deployments, enable connection to an external Kafka cluster by editing the prconfig.xml file. An instance of StreamServiceModule class reads the stream configuration parameters and sets system properties through the System.setProperty() class. Then, the Stream service provider reads the system property configuration.

Note: For cloud deployments, this task is automatically managed for you by Pega Cloud Services.

Before you begin

Download the prconfig.xml file. For more information, see Changing node settings by modifying the prconfig.xml file.

Configuring the Kafka connection

Modify a number of stream-related properties to manage your connection to an external Kafka cluster.

  • services/stream/provider – Denotes the stream service provider type (case sensitive). The default value is Internal. Set this value to ExternalKafka to connect to an external cluster.

  • services/stream/broker/url – Broker URL.

  • services/stream/encryption/security/protocol – Security protocol. Available values include plaintext, ssl, sasl_plaintext, sasl_ssl. The default value is plaintext.

  • services/stream/encryption/truststore/path – Trust store path.

  • services/stream/encryption/truststore/password – Trust store password.

  • services/stream/encryption/keystore/path – Key store path.

  • services/stream/encryption/keystore/password – Key store password.

  • services/stream/encryption/key/password – Key password.

  • services/stream/encryption/sasl/mechanism – Simple Authentication and Security Layer (SASL) mechanism. Kafka supports the GSSAPI(Kerberos), OAUTHBEARER, SCRAM, and PLAIN mechanisms.

  • services/stream/encryption/sasl/jaas/config – SASL Java Authentication and Authorization Service (JAAS) configuration. Escape nested quotes by using the " phrase in the XML file, as demonstrated in this example:

    <env name="services/stream/encryption/sasl/jaas/config" value=" required
    username=&quot;ickafka&quot; password=&quot;6d3300e93b17c4a0b19ed7f0fd9b3658179030bc411a860d7234d0b751e4d3be&quot;;"/>
  • services/stream/category/retention/{categoryName} – Custom retention period for the specified category, in hours. Enter the {categoryName} setting by using lowercase letters in the original stream category name. For example, you can set the retention period for the streams in the QueueProcessor category by configuring this setting to services/stream/category/retention/queueprocessor

    Note: The expected value of this setting is an integer that denotes the hours for retaining the messages in the stream. If you configure this setting to an invalid value or the value is blank, then the default value of60 hours (2.5 days)is used as the retention period. 

External Stream service monitoring

After you configured an external Kafka cluster to manage the Stream service in your application, the node status information is no longer available on the Stream Service landing page. However, you can track the status of the service through the JMX console, as shown in the following example:

"JMX properties"
Monitoring the external Stream service through the JMX console


Did you find this content helpful? YesNo

100% found this useful

Have a question? Get answers now.

Visit the Collaboration Center to ask questions, engage in discussions, share ideas, and help others.

Ready to crush complexity?

Experience the benefits of Pega Community when you log in.

We'd prefer it if you saw us at our best. is not optimized for Internet Explorer. For the optimal experience, please use:

Close Deprecation Notice
Contact us