Editing Load Properties in File Data Point
To change the settings for data load, click Load Properties tab.
In the canvas from the Load Type drop-down list, you can select Avro or File, and the default load type is Avro. When you change the default load type, a reset icon appears beside the drop-down list, and to revert click the reset icon.
Based on the requirements, you can do the following:
- Change the default Property values, then a button appears beside the changed value and to revert click Reset to Default button.
- To rest all the values and set to accommodate general industry standards and data formats, click Reset All to Default link.
Note: To view specific property details, you can enter the keyword in the search bar, and the particular property details are displayed.
Load type: AVRO
| Property | Description | Default Value | Other possible values |
|---|---|---|---|
| Name Space | Can generate desired ‘namespace’ into Avro schema file | ||
| File Name | Specifies the name of the Avro file | instNm.dat | |
| Schema Name | Can generate desired ‘name’ into Avro schema file. | ||
| File Location | Specifies the location of the Avro file | $$SrcFileDir | $$TgtFileDir |
| File Split | Select Yes if you want Split the file else select No | No | YES |
| Split File On (Displays only when File Spilt is "yes") | Select whether to split the file based on size or record count | Size | Record count |
| No of records per file | Specifies the number of records per each file | 1000 | |
| Max Size Per File in KB (Displays only when File Spilt is "yes") | Specify the size of each file when Split File ON property is enabled | 1024 | |
| Create Avro Schema File | Specifies whether to create Avro schema file or not | No | YES |
| Schema File (Displays only when Create Avro Schema File is "yes") | Specifies the file location along with path | ||
| Create Trailer File | Asks whether to create trailer file or not | No | YES |
| Trailer File Template (Displays only when Create Trailer File is "yes") | Specifies the Trailer File template | Displays the file template | |
| Trailer File | Specify to maintain the details of splitted avro files and schema file details like file path, count and total no. of records in each splitted file. | ||
| Compression Codec | Select the compression codex from XZ, Snappy, bzip2 or None | None | bzip2 snappy deflate xz |
| Compression Deflate Level (Displays only when Compression Codec is "deflate" or "xz" ) | Specifies the compression deflate level | -1 | |
| Bad Directory | Specifies the directory name where all Bad Records encountered while uploading data to Avro are stored | $$BadFileDir | |
| Bad Records Threshold | Specifies number of bad records allowed in the bad directory for a successful Data Flow | 0 | |
| Feedback Batch Size | Set the batch size | 1000 | |
| File Format | Specifies the type of file | avro | |
| Generate Checksum? | Used to provide some assurance that a downloaded file has not been altered. | No | YES |
| Checksum Algorithm (Displays only when Generate Checksum is "yes") | To generate checksums by using Algorithm (MD5,SHA1,CRC32..e.t.c). | MD5 | SHA1 SHA256 SHA512 CRC32 |
| Writer Encoder | Specify the encoding format of the system. | UTF-8 | ISO-8859-1 |
| Sync Interval | The stream is flushed by default at the end of each synchronization interval, syncInterval - the approximate number of uncompressed bytes to write in each block | 64000 |
Load type: File
| Property | Description | Default Value | Other possible values |
|---|---|---|---|
| File Name | Specify name of the file | instNm.dat | |
| File Type | Specify type of the file | Delimiter | Fixed Width Excel |
| Is Decryption Required? | Specify this option if the source is encrypted file and want to decrypt the data before loading | NO | YES |
| Enable Uncompress? | Specify this option if the source is compressed file and want to uncompress the file before loading | YES | NO |
| Include Header In File | Specify if the header needs to be included in target file. | NO | YES |
| Create Layout File | Specify the layout format to be created | NONE | CSV, AVSC |
| Layout File Name (Displays only when Create Layout File option is chosen as CSV or AVSC) | Specifies the default layout file name for the chosen CSV or AVSC option. | instanceNm_layout.avsc |
Note: The changes are purely based on the architectural requirements.