File Data object corresponds to structure of delimited and fixed width flat files. The delimited flat file can also be an excel. The file data object can be associated with file server data points, like, file data point, rest data point, GCS data point, AmazonS3 data point, Hadoop DFS data point, JMS data point or Kafka data point. For more details on these data points, refer Working with Data Point.

To work with File Data Object, follow below steps:

Step I: Edit, create or import a File data object

It is recommended to create data object by importing the structure from the data file or schema file.

Step II: Configuring attributes

The File Data Object contains following fields. These fields can be edited by clicking on the field and inputting the required value.

  • Attribute: Displays the name of the field at a certain position in the file. This is a text field and name can be defined as needed. For file data object the name of the field is immaterial. These field names will be used when this data object is used to create data objects of other types.
  • Data Type: Displays the datatype associated with the field. This is a list of applicable data types. You can select appropriate type for the field.
  • Precision: Displays the precision for the varchar, number and decimal data type. This is a text field and name can be defined as per the data in the file.
  • Scale: Displays the scale for the number and decimal data type. This is a text field and name can be defined as per the data in the file.
  • Data Sensitivity: This is relevant when data object is used as a source instance. Specify if an attribute has sensitive information and the data in source system is masked. This is required to be specified when the datatype provided in the data object is only indicative of unmasked data and the target structure should map to the datatype for the masked data. You can map the data type of data sensitive attributes to a common datatype in Diyotta Admin's datatype mapping page. For more information, refer Editing Data Type Mapping. The drop down provides the category of sensitivity of the data. Below are the categories.
    SPIRSH-Sensitive Personally Identifiable Information with a Real Risk of Significant Harm
    SPII-Sensitive Personally Identifiable Information
    PIIPD-Personally Identifiable Information in the public domain
    PII-Personally Identifiable Information
    PD-Personal Data
  • Not Null: Displays if the field can have NULL values. This is a checkbox and can be defined as per the data in the file. For file data object this field is immaterial. This property will be used when this data object is used to create data objects of other types.
  • Key Type: Displays if the field is key field. This is a dropdown with values Primary Key and Foreign key. Select one of these if the field is one of these keys. For file data object this field is immaterial. This property will be used when this data object is used to create data objects of other types.
  • Description: Displays any details provided for the field. This is a text field and you can enter any detail as need.

Note:
  • Following operations are allowed on the database entries: Add, Cut, Copy, Paste, Up, Down, Delete, and Search.
  • From the list of attributes, multiple attributes can be selected and we can perform/apply these operations.
  • To add a new attribute, click Add. By default a new attribute is populated in the last row. If you want to add an attribute at a specific position then, select the attribute in the position prior to it and click Add.
  • To search for a specific attribute, enter the keyword in the search bar, and the page displays the related attributes.

Step III: Configuring properties

The properties tab displays the object level details relevant to extract the data from file.

The grid displays following fields,

PropertyDescriptionDefault valueOther possible values
File TypeDefines the format of extracted data.Delimiter 

Fixed Width

Excel

Column Delimiter (Displays only when File Type is "Delimited" )Specifies the delimiter used between the columns in the file.ASCII 01 (SOH)Any ASCII character
Row Delimiter (Displays only when File Type is "Delimited" )Specifies the delimiter used to mark the end of the row in the file.\\n (New Line Character)Any ASCII character
Rows to SkipSkip number of rows mentioned

File NameSpecify name of the fileinstNm.dat
Escape CharactersThe character immediately following the escape character is escaped.This needs to be specified if the text qualifier is provided and the text qualifier character can appear in the source data. Empty\\ (Recommended)Any ASCII character
Text Qualifier (Displays only when File Type is "Excel" )Specify if the text columns in the source data needs to be enclosed in quotes.


Step IV: Save the changes

To save the changes made to the data object, refer Saving Data Object.

Note: 

Viewing sample Data:

Displays preview of data from the associated object in the database.

Following options are available in View Data tab:

  • By clicking Refresh, the data displayed is reloaded.
  • By clicking Export, the displayed records are exported and saved as excel to default download folder in the machine.
Note: To view, write, and run a SQL query on the Data Object, refer Working with SQL Editor.