Editing Extract Properties in BigQuery Data Point
To change the configuration for data extraction, click Extract Properties tab.
In the Canvas, Extract Type is set to the default extraction type BigQuery API.
- The extract properties can be set one time in the Admin module based on the company standards. Refer the page Editing Extract Properties for more details.
- The changes made to the properties in the data point automatically reflect to all the objects connected to this data point.
- Any extract properties that have been overridden in the dependent objects will not be affected by this change.
- During the execution of the job, the load command properties are defined automatically based on the values provided for the extract properties.
Based on the requirements, you can do the following:
- Change the default Property values, then a button appears beside the changed value and to revert click Reset to Default button.
- To rest all the values and set to accommodate general industry standards and data formats, click Reset All to Default link.
Note: To view specific property details, you can enter the keyword in the search bar, and the particular property details are displayed.
Extraction type: BigQuery API
Property | Description | Default value | Other possible values |
|---|---|---|---|
Extraction Mode | The extraction mode can either be File or Pipe. Pipe option is used to stream the data directly from Source to Target. When File option is used then the data extracted from source is staged on the agent server before loading into the target. | File | Pipe |
File Type | Defines the format of extracted data. (Only delimited text format is supported) | Delimiter | JSON |
LF in String (Displays only when File Type is "Delimiter ") | Helps to handle new line character in source data when target load is not intended with new line character. When enabled, while extracting data, the ASCII 10 (LF) character is replaced with ASCII 5 (ENQ) and ASCII 13 (CR) character is replaced with ASCII 6 (ACK) using respective database functions. | ON | OFF |
Column Delimiter (Displays only when File Type is "Delimiter ") | Specifies the delimiter on the extracted file data. | ASCII 01 (SOH) | Any ASCII character |
Compression | Use this option to specify if the data needs to be compressed before transfer from source agent to Target load agent. | NO | YES |
Compression Type (Displays only when Compression is "Yes") | On choosing a compression format, the user is required to give the corresponding compression codec and compression type (Only default Compression Type is supported). | gzip | |
Encryption | Use this option to specify if the data needs to be encrypted before transfer from source agent to Target load agent. This option is also used when the target is file system and the generated file is required to be encrypted. This encryption is not dependent on source and target systems using same or different agent. For detailed information refer Setting up encryption keys at Agents. | NO | YES |
Encryption Type (Displays only when Encryption is "Yes") | When encryption is enabled then, user needs to specify the encryption algorithm to be used. AES can be used for symmetric key encryption. Choose from 128, 192 and 256 bit key sizes. | AES-128 | AES-192 AES-256 |
| Cipher Operation Mode (Displays only when Encryption is "Yes" and Encryption Type is "AES") | Specify the mode of operation, you can either use CBC (default mode) or GCM. In CBC mode, each block of plaintext is dependent on previous ciphertext block before being encrypted i.e block processing and where as in GCM its parallel processing and independent. | CBC | GCM |
Encrypted AES SecreteKey (Displays only when Encryption type is "AES") | Specify if the AES key is generated by using secret key or not. | NO | YES |
Row Delimiter (Displays only when File Type is "Delimiter ") | Specifies the character to be used to indicate the end of the row in the extracted data. | \\n (New Line Character) | Any ASCII character |
Null Value (Displays only when File Type is "Delimiter ") | Specifies what string literal should indicate the null value in the extracted data. During the data load the column value matching this string will be loaded as null in the target. | NULL | Any string literal NA |
Text Qualifier (Displays only when File Type is "Delimiter ") | Specify if the text columns in the source data needs to be enclosed in quotes. | Empty | Single Double |
Escape Character (Displays only when File Type is "Delimiter ") | The character immediately following the escape character is escaped. This needs to be specified if the text qualifier is provided and the text qualifier character can appear in the source data. | Empty | \\ (Recommended) Any ASCII character |
| Date Style (Displays only when File Type is "Delimiter ") | Specifies how to interpret the date format | YMD | MDY,DMY,DMONY,MONDY,Y2MD, MDY2, DMY2, DMONY2 and MONDY2 |
Date Delimiter (Displays only when File Type is "Delimiter ") | Specifies the separator used in the date format. (Only default delimiter is supported) | - | |
Time Style (Displays only when File Type is "Delimiter ") | Specifies the format of the time portion in the data. | 24HOUR | 12HOUR |
Time Delimiter (Displays only when File Type is "Delimiter ") | Specifies the character used as separate the time components. (Only default delimiter is supported) | : | |
Decimal Delimiter (Displays only when File Type is "Delimiter ") | Specify the decimal delimiter for float/double/numeric data types. | EMPTY | . , |
Fetch Size (Displays only when File Type is "Delimiter ") | Specify after extracting how many rows does the feedback need to be provided in the monitor logs. | 1000 | 10 100 100000 Any numeric value |
| Allow Large Result Set (Displays only when File Type is "Delimiter ") | If set to true allows the query to produce arbitrarily large result tables at a slight cost in performance | FALSE | TRUE |