All the target instance transform corresponding to the Avro databases have similar properties and configuration. 

To work with Avro target instance transform in data flow, follow below steps:

Note:

Selecting the Avro Data Object to be added as target instance transform

Option I: Selecting the Avro Data Object to be added as target instance transform

In the data flow canvas move to Data Flow pane and navigate to Target menu. Here, you can either select or drag and drop the Avro target to canvas.

If the Avro data objects are available then the Create New Avro Target window prompts you to select a Avro data object else 'No Data Object exists in Avro' message is displayed.

  • To search for a specific project, enter the keyword in the Project search bar, and the drop down displays the search result list. Select the required project and click Ok.
  • Select the Avro data object from local object else if there is any Global project that user has access to, then the Project drop-down will list them. Select the Global project from the list if the data object must be chosen from there.

Note: The Data object to be used as target should already be available in Diyotta for it to appear in the list. For creating or importing the data object, refer Creating new data object.

After selection, click OK and the data flow canvas displays the selected Avro data object as target instance.

Note: Upon selecting database type as AVRO, associated file data points are displayed in the data points list.

Link the source instance transform to Avro target instance transform.

Option II: You can also create a target, by using Create As Target option.

In the canvas, select the source data object and click ellipses, a drop down lists available options, from the list select Create As Target.

Likewise, you can also select the option Create As Target from Actions menu.

In both the scenarios, Create As Target window appears and prompts you to provide the details for Avro target instance transform and upon confirmation the target instance transform appears in the canvas. For more information, refer Creating a target from transform in Data flow.

After selection, click OK and the data flow canvas displays the selected Avro data object as target instance.

Configuring Avro target instance transform 

In the General Tab edit the basic details associated with the Avro target instance transform.

Note: In general you can only view Source/Target properties and Script tabs but in target instance by default you can also view Data tab used for debugging purpose.

1. The Name field auto populates the target instance name which is prefixed with TGT and the name is editable.

2. In the Description text-box, provide a description and is optional.

3. The Data Object Name displays the associated data object name and upon clicking the associated name you are navigated to respective data object canvas.

  • To change the associated data object, click Change, then the Change Target data Object window appears and lists the Avro data objects. From the list, click on required data object and click OK.

  • To search for a specific project, enter the keyword in the Project search bar, and the drop down displays the search result list. Select the required project and click Ok.
  • If there is any Global project that user has access to, then the Project drop-down will list them. Select the Global project from the list if the data objects need to be chosen from there.

4. The Data Point field displays the associated data point with the source instance transform and when you click the associated name you are navigated to respective data point canvas.

5. The Transient Object field displays whether its a transient object or not.

6. The Database Type field displays the type of database to which the target is associated 

Viewing the Attributes in target Instance Data Flow

You can view the attributes that are in the associated data object. The attributes listed are not editable. The attributes can be edited only in the associated data object. 

Mapping the Attributes in target Instance Data Flow

In the grid, the attributes in the associated data object are displayed under Target Attribute and you can either manually or automatically map these to attributes from the connected transforms. The mapped attributes are displayed under the Source Attribute. 

Manual mapping: You can manually map the attributes to the target attribute by selecting it from the source Attributes drop down against the specific target attribute. The list will display all the attributes that can be mapped.

Auto mapping: You can automatically map all the target attributes by selecting the Auto Map icon. All the target attributes are mapped by matching the name of the target attribute with the attributes in the linked transforms. If there are multiple transforms prior to the target then the attribute with the same name and which is closest to the target instance will be mapped. If there are no attributes that match the name of the target attribute then the source attribute corresponding to it will remain blank. This will have to be mapped manually.

To modify or remove a mapping, you can select a specific attribute and click Unlink target attribute icon.

You can also unmap all the associated source attributes by clicking Unlink all target Attributes icon.

Configuring the load type

You can specify what type of operation needs to be performed when loading the data into the target table. Choose the operation from the Load Type drop-down.

Note: 

  • The load type drop-down will list only Insert operation if the target instance transform is external to the data flow.
  • For native target instance transforms, the load type will display Insert, Update, Upsert, Delete, and SCD operations.

Load Type: Insert

To insert the data into the target table based on the mapped attributes, select the load type as Insert. During execution the Insert Into statement is generated against the target instance transform. If a target attribute is not mapped in the attribute mapper then, in the insert statement the attribute is excluded and database NULL value will be loaded into it.

You can view the generated SQL in Script tab.

Editing Extract Properties in Target Instance

The extract properties are displayed only when the hive target instance is external.

To change the extract properties for the Avro target, click Extract Properties tab.

By default these properties are set to recommended/default values from data point and the values can be overridden here. To work with extract properties of specific data flow native data point type, refer Working with Data Point and view the extract properties for the specific data point.

Note: To revert the changes to the default values, click Reset All to Default.

Editing Load Properties in File Target Instance

The load properties will be displayed only when the target instance is external and the data flow is not generic. The load properties will correspond to the native data point used to create data flow.

To change the load properties for the file target, click Load Properties tab.

By default these properties are set to recommended/default values from the associated data point and the values can be overridden here. To work with load properties, refer Editing Load Properties in File Data Point. 

Note: To revert the changes to the default values, click Reset All to Default.