Changes to the architecture of the Data Flow service
Valid from Pega Version 8.4
In Pega Platform™ 8.4, the architecture of batch and real-time data flows uses improved node handling to increase the stability of data flow runs. As a result, there are fewer interactions with the database and between the nodes, resulting in increased resilience of the Data Flow service.
If you upgrade from a previous version of Pega Plaftorm, see the following list for an overview of the changes in the behavior of the Data Flow service compared to previous versions:
- Updates to lifecycle actions
- Starting a run
- Triggering pre- and post-activities
- Selecting a node fail policy
- No service nodes and active runs
Nodes no longer communicate and trigger each other, but run periodic tasks instead. As such, triggering a new run does not cause the service nodes to immediately start the run. Instead, the run starts a few seconds later. The same applies to user actions such as stopping, starting, and updating the run. The system also processes topology changes as periodic tasks, so it might take a few minutes for new nodes to join runs, or for partitions to redistribute when a node leaves a run.
To make lifecycle actions more intuitive, the Stop action consolidates both the Stop and Pause actions. The Start action consolidates both the Resume and Start actions.
You can resume or restart stopped and failed runs with the Start and Restart actions. The Start action is only available for resumable runs and continues the run from where it stopped. The Restart action causes the run to process from the beginning. Completed runs can only be restarted. If a run completes with failures, you can restart it from the beginning, or process only the errors by using the Reprocess failures action.
New data flow runs have the Initializing status, and start automatically. You no longer need to manually start a new run, so the New status is now removed.
If there are no nodes available to process a run, the run gets the Queued status and waits for an available node.
The system now triggers pre-activities on a random service node, rather than on the node that triggered the run.
The system triggers post-activities only for runs that complete, fail, or complete with failures. If you manually stop a run with the Stop action, the post-activity does not trigger. However, restarting the run with the Restart action triggers first the post-activity, and then the pre-activity.
You can no longer choose to run pre- and post-activities on all nodes.
For resumable runs, you can no longer select a node fail policy. If a node fails, the partitions assigned to that node automatically continue the run on different nodes.
For non-resumable runs, you can choose to restart the partitions assigned to the failed node on different nodes, or to fail the partitions assigned to the failed node.
If the last data flow node for an in-progress run fails, the run remains in the In Progress state, even if no processing takes place. This behavior results from the fact that data flow architecture now prevents unrelated nodes from affecting runs.
Valid from Pega Version 7.1.4
This release focused on improving the user experience of creating new rules and tightening up the use of specifications in the Case Designer. The layout of the New and Save As forms were made more intuitive and intelligent. Specification actions were made to be more consistent across various aspects of the Case Designer.
- After removing specification references, users can create and save a new specification.
- Developers may now choose either an existing specification or create a new one for subflows and Split-For-Each shapes.
- If a specification is not associated with flow shapes such as sub flows, Utilities, or Split-For-Each, it is possible to import speciications with the same names, and the flow shapes will default to using those imported specs.
- In the outline view, specification changes will be preserved when switching from tab to tab.
- The View Full Details functionality from either Configure Step or the Outline View will display all details, including the specification requirement links.
Valid from Pega Version 7.1.4
This release contains a number of fixes that improve the configuration and execution of Batch decisions – specifically, a number of usability improvements and addresses some issues when creating strategies on IE8.
- Batch Decision functionality has been enhanced in the NBAM environment.
- In NBAM, users can run the strategy execution batch based on the Seed list class instead of the customer class.
- Interaction data will now work even if Action or Organization dimensions do not have values set.
- Running a strategy which includes a Data Import of a structured page reference will work in batch mode.
- Security was enhanced on the VBD Planner.
- The Access of Role to Object rules have been enhanced to allow users to create new Dimension Operator records.
- The NBAM Segment On Canvas will now fetch the SegMap data, even when the generated extended segment class has not been added to the Pages and Classes of the strategy. The SegMap property will (if it exists) be added to a list of properties to fetch, even if the strategy indicates that it is not used.
- When a strategy is defined on a customer class, users may now specify the list of required properties to fetch for the input definition.
- When creating a new strategy, there is now a section to add a Strategy Results class like Business Issue or Group.