File data point is used to configure connectivity to the file servers. The file data point can be used to access files on Diyotta agent server or on remote locations like, SFTP, FTP and Amazon S3 reachable from the agent server. For each local server or remote server, a separate data point will have to be created. Any location on the server can be accessed using the Data point as long as the user has necessary privileges. The file data point will be associated to any file object created like, Delimited, Fixed width, XML, JSON, etc. 

To work with File Data Point, follow below steps:

Step I: Create a New Data Point

Step II: Provide connection details

To connect to File data point following details need to be provided in the Properties tab.

1. File Location: Specify the base location of the file on the server.

To search for a specific file in the list, from File Location drop down list in the search box enter the search criteria and the canvas displays relevant files list.

  • By default the drop down will show system parameters corresponding to the location on the Agent installation

$$SrcFileDir - $DIYOTTA_HOME/agent/app/source
$$TgtFileDir - $DIYOTTA_HOME/agent/app/target
$$ArchFileDir - $DIYOTTA_HOME/agent/app/archive

  • Change this location to point to where file will be placed.
  • During execution this path will be appended to the file name specified in the extraction properties

2. Remote server details: If the files are located on remote server then, select the check box Files on Remote Location?

    Drop down with supported remote server connection will be displayed. Select the required remote server connection.

a). FTP: To work with FTP remote location, refer Working with FTP Data Point.

b). SFTP: To work with SFTP remote location, refer Working with SFTP Data Point.

c). Amazon S3: To work with Amazon S3 remote location, refer Working with Amazon S3 Data Point.

Note: All the fields in the Properties tab can be parameterized using project parameters. To parameterize the fields, refer Working with Project Parameters.

3. Assign AgentTo assign or change the associated agent click Change. The Change Agent window appears and displays the list of available Agents. From the list, select the required Agent Name.

Note: To search for a specific agent, enter the keyword in the search bar, and the window displays the search result list. Select the required agent and click Ok.
  • If Default agent is assigned to the Project then automatically, the Default agent will be associated with the new Data point created.
  • If Default agent is not assigned to the Project then, no agent will be assigned automatically and and appropriate agent needs to be assigned to the data point.
  • The firewall needs to be opened from the Agent server to the File server and user specified to connect should have appropriate privilege to access the path where file will be placed.

Step III: Save the data point 

Note:

Step IV: Modify the configured Extract and Load properties 

When moving data from one system to another the data is extracted from the source system, moved over the network and loaded into the target system. The SQLs and commands generated during execution of the jobs to extract and load data are generated based on the properties defined for these. The properties associated with the extraction and load of data should depend on the format, performance and variety of the data being moved. These properties vary based on the environment and the type of the system. Diyotta comes with default properties that covers most of the known scenarios. 

To modify these properties, refer Editing Extract Properties in File Data Point and Editing Load Properties in File Data Point.

  • The default values for extract and load properties can be configured in the Admin module and these properties reflect in the Studio module.
  • The extract and load properties set in data point are by default used in the source and target instance of the data flow and the job flows.
  • It is a good practice to set the extract and load properties as per the company standards in the data point.
  • However, if needed any specific property can be overridden in the data flow or job flow.