To change the configuration for data extraction, click Extract Properties tab.

In the Canvas, Extract Type is set to the default extraction type BigQuery API. 

  • The extract properties can be set one time in the Admin module based on the company standards. Refer the page Editing Extract Properties for more details.
  • The changes made to the properties in the data point automatically reflect to all the objects connected to this data point.
  • Any extract properties that have been overridden in the dependent objects will not be affected by this change.
  • During the execution of the job, the load command properties are defined automatically based on the values provided for the extract properties.

Based on the requirements, you can do the following:

  • Change the default Property values, then a button appears beside the changed value and to revert click Reset to Default button.
  • To rest all the values and set to accommodate general industry standards and data formats, click Reset All to Default link.

Note: To view specific property details, you can enter the keyword in the search bar, and the particular property details are displayed.

Extraction type: BigQuery API

Property

Description

Default value

Other possible values

Extraction Mode

The extraction mode can either be File or Pipe.

Pipe option is used to stream the data directly from Source to Target.

When File option is used then the data extracted from source is staged on the agent server before loading into the target.

File 

Pipe

File Type

Defines the format of extracted data. (Only delimited text format is supported)

Delimiter 

JSON

LF in String (Displays only when File Type is "Delimiter ")

Helps to handle new line character in source data when target load is not intended with new line character.

When enabled, while extracting data, the ASCII 10 (LF) character is replaced with ASCII 5 (ENQ) and ASCII 13 (CR) character is replaced with ASCII 6 (ACK) using respective database functions.

ON

OFF

Column Delimiter (Displays only when File Type is "Delimiter ")

Specifies the delimiter on the extracted file data.

ASCII 01 (SOH)

Any ASCII character

Compression

Use this option to specify if the data needs to be compressed before transfer from source agent to Target load agent.

NO 

YES

Compression Type (Displays only when Compression is "Yes")

On choosing a compression format, the user is required to give the corresponding compression codec and compression type (Only default Compression Type is supported).

gzip 


Encryption

Use this option to specify if the data needs to be encrypted before transfer from source agent to Target load agent.

This option is also used when the target is file system and the generated file is required to be encrypted. This encryption is not dependent on source and target systems using same or different agent.

For detailed information refer Setting up encryption keys at Agents.

NO 

YES

Encryption Type (Displays only when Encryption is "Yes")

When encryption is enabled then, user needs to specify the encryption algorithm to be used.

AES can be used for symmetric key encryption. Choose from 128, 192 and 256 bit key sizes.

AES-128

AES-192

AES-256

Cipher Operation Mode (Displays only when Encryption is "Yes" and Encryption Type is "AES")

Specify the mode of operation, you can either use CBC (default mode) or GCM. In CBC mode, each block of plaintext is dependent on previous ciphertext block before being encrypted i.e block processing and where as in GCM its parallel processing and independent.

CBCGCM

Encrypted AES SecreteKey (Displays only when Encryption type is "AES")

Specify if the AES key is generated by using secret key or not.

NO 

YES

Row Delimiter (Displays only when File Type is "Delimiter ")

Specifies the character to be used to indicate the end of the row in the extracted data.

\\n (New Line Character)

Any ASCII character

Null Value (Displays only when File Type is "Delimiter ")

Specifies what string literal should indicate the null value in the extracted data. During the data load the column value matching this string will be loaded as null in the target.

NULL 

Any string literal

NA

Text Qualifier (Displays only when File Type is "Delimiter ")

Specify if the text columns in the source data needs to be enclosed in quotes.

Empty

Single

Double

Escape Character (Displays only when File Type is "Delimiter ")

The character immediately following the escape character is escaped.

This needs to be specified if the text qualifier is provided and the text qualifier character can appear in the source data.

Empty

\\ (Recommended)

Any ASCII character

Date Style (Displays only when File Type is "Delimiter ")Specifies how to interpret the date format

YMD 

MDY,DMY,DMONY,MONDY,Y2MD, MDY2, DMY2, DMONY2 and MONDY2

Date Delimiter (Displays only when File Type is "Delimiter ")

Specifies the separator used in the date format. (Only default delimiter is supported)


Time Style (Displays only when File Type is "Delimiter ")

Specifies the format of the time portion in the data.

24HOUR 

12HOUR

Time Delimiter (Displays only when File Type is "Delimiter ")

Specifies the character used as separate the time components. (Only default delimiter is supported)


Decimal Delimiter (Displays only when File Type is "Delimiter ")

Specify the decimal delimiter for float/double/numeric data types.

EMPTY

.

,

Fetch Size (Displays only when File Type is "Delimiter ")

Specify after extracting how many rows does the feedback need to be provided in the monitor logs.

1000 

10

100

100000

Any numeric value

Allow Large Result Set (Displays only when File Type is "Delimiter ")If set to true allows the query to produce arbitrarily large result tables at a slight cost in performanceFALSETRUE