Interesting pattern: OIC and EventHub for highly available short-term cloud buffering

Using OIC and EventHub for highly available short-term persistent cloud buffering

Leveraging highly available cloud solutions, customers are able to offer potentially significant improvements in up-time over an on-premise solution.

However, moving all function to cloud can introduce risk for many customers. Using this pattern, customers can provide useful functionality to external data providers and consumers, whilst still maintaining a primary on-premise footprint.

Overview

OIC and EventHub are used together in cloud, to provide a staging area for asynchronous external data providers, that is highly available. The on-premise system queries EventHub from on-premise on a periodic basis to retrieve pending work items for processing.

Key Assumptions

  • The workloads are asynchronous in nature.
  • The on-premise system is significantly less available than the cloud system

overview

Basic implementation

  • OIC is configured to receive messages from external actors, using a well known API endpoint. REST is typical.
  • The OIC integration flow posts the message into an EventHub topic.
  • On a periodic basis, depending on availability, the on-premise system queries EventHub for pending work.
  • The work is retrieved from the EventHub topic.
  • The work is forwarded to on-premise systems for further processing.

The on-premise consumer

The on-premise system would typically use an existing packaged solution for consumption from the EventHub system. There are many implementations of a consumer in various languages depending on customer need. One could also use the EventHub REST API to consume messages, though that is more complex than publishing. The EventHub consumer API is typically internet accessible. As such, there is no need for any on-premise agent or other element to get access to it.

Notes on EventHub

EventHub is based on Kafka, a message delivery platform with a very highly available design. It is not a queue implementation, though it superficially resembles one. One key difference is messages are always added for a specific lifetime, and EventHub aggressively enforces this lifetime. Typically, the lifetime is measured in hours, not days.

Consumers from EventHub have much more control over how they consume messages than typical queuing systems, which is both a positive and a negative, as it can introduce significant complexity in a solution that requires consumption from EventHub. In this scenario, that would be accomplished by the on-premise system, typically via an API implementation.

OIC as subscriber

This post addresses the need for an endpoint that can asynchronously forward in a reliable fashion. As such, OIC is acting as a publisher to EventHub. It is possible to enable OIC to listen to an EventHub topic. That will be discussed in a followup blog post. This will be useful for those looking for a REST API based solution in general.

Add Your Comment