MongoDB is a document-oriented NoSQL database used for high volume data storage. Instead of having data in a relational type format, it stores the data in documents. This makes MongoDB very flexible and adaptable by using uses JSON-like documents with schemata. Each MongoDB contains collections which in turn contains documents. Each document can be different with a varying number of fields. The size and content of each document can be different from each other.

To work with MongoDB Data Point, follow below steps:

Step I: Create a New Data Point

Step II: Provide connection details

1. To connect to MongoDB database following details need to be provided in the Properties tab.

  • Host: Specify the hostname or the IP address of the MongoDB system
  • Port: Specify the port associated to the MongoDB system
  • App User: Specify the user id that will be used to connect to the MongoDB system. This user should have necessary privileges to access the data in the databases
  • Password: Specify the password for the App User.To use the project parameter for the password, check the Use Project Parameters option, and you can view and select the required Project Parameter from the Password drop-down. 
  • Jdbc Options: Specify the options that should be used along with JDBC URL to connect to MongoDB.
    For example, following details are provided in JDBC Options to connect to MongoDB: user=diyotta, password=****, db=TEST_DB.
  • Database: Collection of information that is organized so that it can be easily accessed, managed and updated.
  • Row Scan depth : The maximum number of rows to scan to look for the columns available in a table. 

2. If SSL is Enabled following details need to be provided in the Properties tab:

  • Trust Store: Specify key store path which is generated by client.
  • Trust Store Password: Specify password for trust store which is created while generating trust store file. 
  • Key Store: Specify trust store path which is generated by server.
  • Keystore Password: Specify password for key store which is generated by client which is created while generating key store file.
  • SSL CA File: Specify the SSL CA file (.pem file) name which is containing the root certificate chain which is generated by client.
  • SSL PEM Key Password: Specify the password to decrypt the SSL CA File.
  • SSL PEM Key File: Specify the SSL CA File name containing the certificate and key.
  • SSL CRL File: Specify the SSL CRL File name containing the certificate revocation list.
  • SSL Allow Invalid Certificates: Enable this option to bypass the validation for server certificates.
  • SSL Allow Invalid Hostnames: bypass the validation for server name.
  • SSL FIPS Mode: Enable this option to use FIPS mode.

Note:

  • Mandatory field names are suffixed with *. To establish the connection, provide all the mandatory property field values.
  • All the fields in the Properties tab can be parameterized using project parameters. To parameterize the fields, refer Working with Project Parameters.

3. Assign Agent: To assign or change the associated agent click Change. The Change Agent window appears and displays the list of available Agents. From the list, select the required Agent Name.

Note: To search for a specific agent, enter the keyword in the search bar, and the window displays the search result list. Select the required agent and click Ok.
  • If Default agent is assigned to the Project then automatically, the Default agent will be associated with the new Data point created.
  • If Default agent is not assigned to the Project then, no agent will be assigned automatically and appropriate agent needs to be assigned to the data point.
  • When connecting to the Agent server then, the agent installation user should have appropriate privilege to access the path where file will be placed.
  • When connecting to the remote server then, the firewall needs to be opened from the Agent server to it and user specified to connect should have appropriate privilege to access the path where file will be placed

Step III: Test the data point connection

  • To validate that the data point is able to connect to the MongoDB data point database using the details provided, refer Test Data Point Connection. 

Step IV: Save the data point 

Note:

Step V: Modify the configured Extract properties

When moving data from one system to another the data is extracted from the source system, moved over the network and loaded into the target system. The SQLs and commands generated during execution of the jobs to extract and load data are generated based on the properties defined for these. The properties associated with the extraction and load of data should depend on the format, performance and variety of the data being moved. These properties vary based on the environment and the type of the system. Diyotta comes with default properties that covers most of the known scenarios. 

To modify these properties, refer Editing Extract Properties in MongoDB Data Point.

  • The default values for extract properties can be configured in the Admin module and these properties reflect in the Studio module.
  • The extract properties set in data point are by default used in the source and target instance of the data flow and the job flows.
  • It is a good practice to set the extract properties as per the company standards in the data point.
  • However, if needed any specific property can be overridden in the data flow or job flow.