Twitter Data Point is used to configure connectivity to Twitter database based on access token authorization.

To work with Twitter Data Point, follow below steps:

Step I: Create a New Data Point

Step II: Provide connection details

1. To connect to Twitter database following details need to be provided in the Properties tab.

  • Authentication Type: Specifies the type of authentication that is required to connect to Twitter web service. By default, the authentication type is set to OAuth 2.0 and is not editable.
  • Access Token: Access Token is automatically populated once you refresh the datapoint after establishing connection or after doing Test Connection. If you have already obtained access token externally, then you may need not go through the OAuth flow by doing test connection, you can just put the access token and save the data point.
  • Batch Size: Specify the number of batches to be processed in memory and loaded at a time. If the value specified is zero then, all batches are extracted at once.
  • Time Out: Specify the time out duration in minutes. By default, it is set to 0 which means no specified time out duration.
  • Access Token Secret: Access Token Secret is automatically populated once you refresh the datapoint after establishing connection or after doing Test Connection.
  • OAuth Client ID: Specify the API Key of the application registered with twitter. To access the twitter api, login to https://dev.twitter.com/apps and create a new application.
  • OAuth Client Secret: Specify the API Secret Key of the application registered with twitter. To access the twitter api, login to https://dev.twitter.com/apps and create a new application.
  • CallbackURL: Secured domain URL of the repo which is used to register in.
    • If you are making a Web app, set the Callback URL to the URL you would like users to be returned to after they have granted your application privileges.
    • If you are making a desktop application, set the Callback URL to http://localhost:33333/
  • JDBC Options: Specify the options that should be used along with JDBC URL to connect to Twitter. For example, following details are provided in JDBC Options to connect to Twitter: user=diyotta, password=******, db=TEST_DB.

  • Mandatory field names are suffixed with *. To establish the connection, provide all the mandatory property field values.
  • Upon enabling Get Access Token check box, the canvas displays OAuth Client ID, OAuth Client Secret, and Callback URL fields.


2. Assign Agent: To assign or change the associated agent click Change. The Change Agent window appears and displays the list of available Agents. From the list, select the required Agent Name.

Note: To search for a specific agent, enter the keyword in the search bar, and the window displays the search result list. Select the required agent and click Ok.

  • If Default agent is assigned to the Project then automatically, the Default agent will be associated with the new Data point created.
  • If Default agent is not assigned to the Project then, no agent will be assigned automatically and appropriate agent needs to be assigned to the data point.
  • When connecting to the Agent server then, the agent installation user should have appropriate privilege to access the path where file will be placed.
  • When connecting to the remote server then, the firewall needs to be opened from the Agent server to it and user specified to connect should have appropriate privilege to access the path where file will be placed.

Step III: Test the data point connection

  • To validate that the data point is able to connect to the Twitter data point database using the details provided, refer Test Data Point Connection.

Step IV: Save the data point 

Note:

Step V: Modify the configured Extract properties

When moving data from one system to another, the data is extracted from the source system, moved over the network and loaded into the target system. The SQLs and commands generated during execution of the jobs to extract the data are generated based on the properties defined for these. The properties associated with the extraction of data should depend on the format, performance and variety of the data being moved. These properties vary based on the environment and the type of the system. Diyotta comes with default properties that covers most of the known scenarios.

To modify these properties, refer Editing Extract Properties in Twitter Data Point.

  • The default values for extract properties can be configured in the Admin module and these properties reflect in the Studio module.
  • The extract properties set in data point are by default used in the source and target instance of the data flow and the job flows.
  • It is a good practice to set the extract properties as per the company standards in the data point.
  • However, if needed any specific property can be overridden in the data flow or job flow.