You can now define File data sets for more advanced scenarios more flexibly with an additional set of customization options.
Custom stream processing
To improve customer data security, apply encryption and decryption algorithms to write and read that data by implementing your own Java classes. This way you can also add compression methods, in addition to the basic .gzip and .zip formats.
Manifest file support
Gain more control over source data by listing the files to read in an .xml manifest file, instead of using a fixed filepath, as in the following example:
<manifest> <files> <file> <name>file0001.csv</name> </file> <file> <name>file0002.csv</name> </file> </files> </manifest>
Meta files created on save
Retrieve detailed information on every file that you save. For every file, the system creates a dedicated meta file which includes the saved file name, size, and the number of records.
Date and time pattern support
To facilitate data management processes, automatically extend the filename of every file with a date and time of creation by adding a Java SimpleDateFormat string to the file path.
Parser configuration on demand
Choose if you want to automatically configure the parser settings for the data in your File data set. This way, even if you update the file path, your parser configuration and the .csv file column mapping do not change.
For more information, see Creating a File data set for files on repositories and Requirements for custom stream processing in File data sets.