To work with GCS  Data Point, follow below steps:

Step I: Create a New Data Point

Step II: Provide connection details

To connect to GCS API following details need to be provided in the Properties tab.

  • Project ID: The identification number given to particular project and it is always unique.
  • Authentication Type: It can be either Service Account or Access Tokens.

1. If you want to do authentication based on service account key file, choose the Authentication Type as Service Account, then following details must be provided:

    • In Service Account Key File, browse and select the Service account key file generated from API libraries in the project. 

2. For Access Token, following details are populated on the screen:

    • Access Token:  If you want to do authentication based on access token generated in Google which is specific to this project, choose the Authentication Type as Access Token, then following details must be provided: 
    • To Obtain Access token from external service, enable this option to use SSL connectivity between Agent and GCS server. Upon enabling this option, following fields are displayed on the screen. Provide details for this to use SSL in connectivity. Service account and Access token
      1. External Service Command: Specify the external command to generate the Access Token
      2. Response Type: Select the response type and by default its JSON.
      3. Access Token Key Name: Specifies the access token field name which is available in generated response JSON file

For both the Authentication Types following details are similar:

  • Bucket Name: Specify the GCS bucket name.
  • Proxy: If you want to connect to external network connection, you can enable Use Proxy option and set the type of proxy you are using, address, and port on which to reach it.
    1. Proxy HostSpecify the Hostname or the IP address of the GCS system
    2. Proxy Port: Specify the port associated to the GCS system
    3. Protocol: It can be either HTTP or HTTPS

Note:

  • Mandatory field names are suffixed with *. To establish the connection, provide all the mandatory property field values.
  • All the fields in the Properties tab can be parameterized using project parameters. To parameterize the fields, refer Working with Project Parameters.

Assign Agent: To assign or change the associated agent click ChangeThe Agent window appears and displays the list of available Agents. From the list, click on required Agent Name.

Note: To search for a specific agent, enter the keyword in the search bar, and the window displays the search result list. Select the required agent and click Ok.

  • If Default agent is assigned to the Project then automatically, the Default agent will be associated with the new Data point created.
  • If Default agent is not assigned to the Project then, no agent will be assigned automatically and appropriate agent needs to be assigned to the data point.
  • When connecting to the Agent server then, the agent installation user should have appropriate privilege to access the path where file will be placed.
  • When connecting to the remote server then, the firewall needs to be opened from the Agent server to it and user specified to connect should have appropriate privilege to access the path where file will be placed.

Step III: Test the data point connection

  • To validate that the data point is able to connect to the GCS data point database using the details provided, refer Test Data Point Connection. 

Step IV: Save the data point 

Note:

Step V: Modify the configured Extract and Load properties 

When moving data from one system to another the data is extracted from the source system, moved over the network and loaded into the target system. The SQLs and commands generated during execution of the jobs to extract and load data are generated based on the properties defined for these. The properties associated with the extraction and load of data should depend on the format, performance and variety of the data being moved. These properties vary based on the environment and the type of the system. Diyotta comes with default properties that covers most of the known scenarios. 

To modify these properties, refer Editing Extract Properties in Google Cloud Storage Data Point and Editing Load Properties in Google Cloud Storage Data Point.

  • The default values for extract and load properties can be configured in the Admin module and these properties reflect in the Studio module.
  • The extract and load properties set in data point are by default used in the source and target instance of the data flow and the job flows.
  • It is a good practice to set the extract and load properties as per the company standards in the data point.
  • However, if needed any specific property can be overridden in the data flow or job flow.