Diyotta’s Data Connect adds real-time data ingestion feature to the Diyotta’s core platform. Built on top of the controller-agent architecture, Data Connect enables organization to perform change data capture (CDC) from RDBMS like Oracle, MySQL, MSSQL and ingest streaming data from JMS based message brokers and Kafka. Under the hood, it has a stack of  trusted technologies such as Kafka, Kafka Connect & Zookeeper. Bring your own enterprise Kafka or use the Diyotta bundled community stack.

The real time large sets of data can be moved from source to target using data connect. Here, the source connectors are used to pull the real time data and loaded to target sink connectors. To process the data connect flow, various source and sink connectors are being utilized. The real time data is loaded directly from source to sink connector by traversing through various steps. 

The steps include selecting source connector, table selection (data objects), selecting sink connector, assigning source and sink properties, reviewing the details and upon submission, the data connect is successfully created.

Below are the supported Sources and Sinks in Data Connect. For more details on these refer Working with Source Connectors and Working with Sink Connectors.

SourcesSinks

JDBC Sources

  • MSSQL
  • Oracle

CDC Sources

  • Oracle
  • MySQL

Streaming Sources

  • Kafka
  • JMS (Messaging)

File System Sinks

  • File
  • DFS
  • Hive

JDBC Sinks

  • MSSQL

Cloud Sinks

  • Redshift
  • BigQuery
  • Google PubSub

Streaming Sinks

  • Kafka

Below chart shows the compatibility of the Sources to Sinks in Data Connect.

Sources

Sinks

FileDFSHiveMSSQLRedshiftBigQueryGoogle PubSubKafka

MSSQL (JDBC)

x

x

x

x


x 

x 

x

Oracle (JDBC)

x



x


x

x

x

Oracle CDC


x

x


x




MySQL CDC


x

x


x




Kafka

x

x

x

x


x

x


JMS (Messaging)

x



x


x

x

x