Data Connector

A repo to periodically reads data from the configured data sources and publish it to the their paired data sinks, at the configurable interval.

This repo currently supports the following Data Source-Sink Pairs:

1. RESTful APIs to MQTT Broker
2. SQL Server to AIO Distributed Data Store (DSS)

Design Overview

Http-MQTT: To understand MQTT topic generation and its customization options please refer to the document here.

SQL-DSS: To understand SQL and DSS customization options please refer to the document here.

Features

1. Configurable polling on RESTful endpoints using GET verb.
2. Configurable Polling on SQL DB endpoints using SQL DQL queries.
3. MQTT and DSS connectivity using anonymous, username/password or SAT token in Azure IoT Operations.

Deploy in K8s

1. Add Helm Repo:

   helm repo add aio-extensions https://raw.githubusercontent.com/suneetnangia/data-connector/release_management

2. Copy configuration file from the example:

   1. For REST-MQTT AIO Connector here.
   2. For REST-MQTT Non-AIO here.
   3. For SQL-DSS AIO Connector here.

3. Update configuration file with your specifics, refer to the links in step 2 above.

4. Install Helm package (appsettings.json content is stored as a K8s secret):

   1. For AIO:

      helm install data-connector-01 aio-extensions/aio-data-connector --namespace azure-iot-operations --create-namespace --set-file appsettingsContent=$pwd./<path to your>/appsettings.json

   2. For non-AIO

      helm install data-connector-01 aio-extensions/data-connector --namespace azure-iot-operations --create-namespace --set-file appsettingsContent=$pwd./<path to your>/appsettings.json

5. Uninstall Helm package (after evaluation):

   helm uninstall data-connector-01 -n azure-iot-operations

Backlog

Please refer to the project board here.

Development Loop