Changes to the architecture of the Data Flow service
Valid from Pega Version 8.4
In Pega Platform™ 8.4, the architecture of batch and real-time data flows uses improved node handling to increase the stability of data flow runs. As a result, there are fewer interactions with the database and between the nodes, resulting in increased resilience of the Data Flow service.
If you upgrade from a previous version of Pega Plaftorm, see the following list for an overview of the changes in the behavior of the Data Flow service compared to previous versions:
- Updates to lifecycle actions
- Starting a run
- Triggering pre- and post-activities
- Selecting a node fail policy
- No service nodes and active runs
Nodes no longer communicate and trigger each other, but run periodic tasks instead. As such, triggering a new run does not cause the service nodes to immediately start the run. Instead, the run starts a few seconds later. The same applies to user actions such as stopping, starting, and updating the run. The system also processes topology changes as periodic tasks, so it might take a few minutes for new nodes to join runs, or for partitions to redistribute when a node leaves a run.
To make lifecycle actions more intuitive, the Stop action consolidates both the Stop and Pause actions. The Start action consolidates both the Resume and Start actions.
You can resume or restart stopped and failed runs with the Start and Restart actions. The Start action is only available for resumable runs and continues the run from where it stopped. The Restart action causes the run to process from the beginning. Completed runs can only be restarted. If a run completes with failures, you can restart it from the beginning, or process only the errors by using the Reprocess failures action.
New data flow runs have the Initializing status, and start automatically. You no longer need to manually start a new run, so the New status is now removed.
If there are no nodes available to process a run, the run gets the Queued status and waits for an available node.
The system now triggers pre-activities on a random service node, rather than on the node that triggered the run.
The system triggers post-activities only for runs that complete, fail, or complete with failures. If you manually stop a run with the Stop action, the post-activity does not trigger. However, restarting the run with the Restart action triggers first the post-activity, and then the pre-activity.
You can no longer choose to run pre- and post-activities on all nodes.
For resumable runs, you can no longer select a node fail policy. If a node fails, the partitions assigned to that node automatically continue the run on different nodes.
For non-resumable runs, you can choose to restart the partitions assigned to the failed node on different nodes, or to fail the partitions assigned to the failed node.
If the last data flow node for an in-progress run fails, the run remains in the In Progress state, even if no processing takes place. This behavior results from the fact that data flow architecture now prevents unrelated nodes from affecting runs.
The UpdateAdaptiveModels agent causes an exception after Pega 7.2 to 7.2.1 upgrade
Valid from Pega Version 7.2.1
After the Pega 7 Platform is upgraded from version 7.2 to 7.2.1, the log files might show an error that is caused by the UpdateAdaptiveModels agent. This agent is enabled by default and is responsible for updating scoring models in the Pega 7 Platform. If you use adaptive models in your solution, you can avoid this error by configuring the Adaptive Decision Manager service. If you do not use adaptive models, disable the UpdateAdaptiveModels agent.
Reconfiguration of the Adaptive Decision Manager service after upgrade to Pega 7.2.1
Valid from Pega Version 7.2.1
After you upgrade the Pega 7 Platform to version 7.2.1, you need to reconfigure the Adaptive Decision Manager service. Beginning with Pega 7.2.1, the Adaptive Decision Management (ADM) service is native to the Pega 7 Platform and is supported by the Decision data node infrastructure.
For more information, see Services landing page.