Service Bus to Data Platform

The Service Bus to Data Platform building block is one of our core building blocks within the integration platform.  It is intended to facilitate a simple route for messages on service bus to end up in the Data Lake from where they can be processed by Spark jobs.

There are 2 ingestion routes into the data lake:

1. Bulk Data ingestion by pipelines querying data from line of business systems on a schedule

2. Messaging ingestion where service bus can route to the data lake.

In this building block you would have a topic that messages are sent to on the service bus and then from there 1 or more subscriptions take data to interfaces for various destination systems who are interested in that message.

You can also add a subscription to the topic which will route the message to a single queue which can process different types of message which is listened to by an Azure Function.  This function processes messages and writes them to an event hub within the data platform and they are then processed using the event hub capture mechanism which writes them to the data lake.

This hands over the responsibility from the integration platform to the data platform.