Skip to main content


         This documentation site is for previous versions. Visit our new documentation site for current releases.      
 

Importing large amounts of data by using the data import File Listener

Updated on December 21, 2020

When you need to import large volumes of data (millions of rows), use the data import File Listener instead of the data import wizard. The data import File Listener uses multithreading for faster throughput, while the wizard uses single-thread processing.

Note: To gain increased performance when importing your data, see Improving data import performance by using configuration templates.

Pega Sales Automation includes a File Listener for the following entities:

  • Operator
  • Contact
  • Household
  • Lead (individual and business)
  • Opportunity (individual and business)
  • Organization
  • Task
  • Customer activity
  • Territory
  • Account

File Listener import recommendations

For best performance while using the data import File Listener, keep the following recommendations in mind:

  • Before beginning to import all of your records, import a few to start with and fix any issues that occur.
  • The size of the File Listener base upload should not exceed 1 million records in a single file.
  • The recommended number of records for batch uploads is 1,000 records. You can set this figure in App StudioSettingsApplication Settings.
  • To improve performance by disabling creation of an audit history, use Add Only mode for the initial data import.
  • To ensure maximum parallel processing, there must be as many input files for the file listener as there are threads, because each thread processes only one file at a time. You can set the number of X thread in the File Listener properties in Dev Studio, in the Listener properties section.
  • In a PostgreSQL single-tenant system, unique ID generation is highly efficient in a high-volume import process. Work item IDs are generated in batches, and you can set the batch size with the idGenerator/defaultBatchSize dynamic system setting. For more information, see Increased performance for work ID generation.
  • In non-PostgreSQL multi-tenant systems, high-volume import process should not include generating unique IDs. Instead, you can save time by including pyID for work object records in the CSV import file to skip calling the GenerateID activity. After the import of contacts is complete, update the unique ID stored in the data-uniqueID database table. Set the Table name value in the data-uniqueID database table to the last imported pyID record in the contact table.
  • Database indexes improve query performance. However, when you update a large database table, the system performs reindexing, which can cause lower performance. Remove non-essential indexes during the import phase. After the import is complete, re-enable indexing.
What to do next: See the section "Preparing the data".

Preparing the data

The data import File Listener uses the same underlying APIs as the data import wizard to process files located in predetermined folders on the server. Importing data by using the data import File Listener requires the use of templates. As a best practice, use the data import wizard to make any template changes prior to using the file listener to import any data. For more information, see Preparing the data and Pega Sales Automation sample data templates.

What to do next: See the "Configuring the File Listener data import for on-premises environments" or the "Configuring the File Listener data import for cloud Pega Cloud" section.

Configuring the File Listener data import for on-premises environments

Before running a data import by using File Listener, you must configure the data import settings.

  1. In the navigation pane of App Studio, click SettingsApplication Settings.
    1. Optional: To modify the default template and purpose configuration, in the navigation pane of Dev Studio, click App, and then search for and open the Resource Settings data transform.
    By default, the system configures the data import File Listener with SA_<name of objects> as a template and Add or update as a purpose.
  2. On the Features tab, in the File Listener Configurations section, enter the path to the root folder in the File Listener source location, and the email address to which you want to send notifications.
  3. Optional: To disable creating an audit history and to improve performance, in the File Listener Configurations section, select the Initial Data Migration check box.

    Note: If you disable creating audit history, after the import completes, clear the Initial Data Migration check box to generate audit records.
  4. Click Save.
  5. Optional: If you want to modify the default template and purpose configuration, in the Dev Studio navigation pane, click App, and then search for and open the ResourceSettings data transform.
    Resource settings example configuration
    Example ResourceSettings data transform configuration.
    Result: By default, the data import File Listener is configured with the SA_<name of objects> template and the Add or update purpose.
What to do next: See the "Running the File Listener data import" section.

Configuring the File Listener data import for Pega Cloud

Before running a data import by using File Listener, you must configure the data import settings.

  1. In the navigation pane of App Studio, click SettingsApplication Settings.

    Note: As a best practice, use an SFTP server implementation.
  2. In the header of Dev Studio, search for and select the storage/class/defaultstore:/type dynamic system setting (DSS).
  3. In the Value field, enter filesystem.
  4. Click Save.
  5. In the header of Dev Studio, search for and select the FileListenerSourceLocation dynamic system setting.
  6. In the Value field, enter the root folder in the File Listener source location.
  7. Click Save.
What to do next: See the "Running the File Listener data import" section.

Running the File Listener data import

After you configure the data import settings, run File Listener to import large volumes of data.

  1. In the navigation pane of Dev Studio, click RecordsIntegrations-ResourcesFile Listener, and then open the listener that you want to run.
  2. In the Listener nodes section, clear the Block startup check box.
  3. In the Source properties section, enter the source file format.
    File Listener only supports csv and txt files.
  4. Click Save.
  5. In the header of Dev Studio, click the Switch Studio menu, and then click Admin Studio.
  6. In the navigation pane of Admin Studio, click ResourcesListeners, and then open the listener that you configured.
  7. In the Requestor login section, enter your user name and password.
Result: After File Listener processes an entire file, output files are created in the source file location that you specified in App Studio. The output file lists failed records and information about each error. The system emails the data import summary to the notification email addresses that you specify in the File Listener configuration process.
  • Previous topic Improving data import performance by using configuration templates
  • Next topic Data import rules and extension points

Have a question? Get answers now.

Visit the Support Center to ask questions, engage in discussions, share ideas, and help others.

Did you find this content helpful?

Want to help us improve this content?

We'd prefer it if you saw us at our best.

Pega.com is not optimized for Internet Explorer. For the optimal experience, please use:

Close Deprecation Notice
Contact us