OIC Scheduler- Decoupled Scheduler and Business Logic Pattern

NOTE: This Blog is retired.

OIC now provides dynamic threading model for scheduled orchestrations. So thread restrictions mentioned in this blog are no more relevant. However, fire & forget pattern is still valid. Integration designers can use approach mentioned in this blog to implement fire & forget pattern in scheduled orchestrations. But ignore the threading model.

Scheduled integration interfaces are widely used in batch integrations. Oracle Integration cloud provides feature to build scheduled integration flows, that can execute at defined frequency (Link). However sometimes users complain about inconsistent behavior of scheduled flows like schedules are getting delayed by few minutes to few hours.

In this blog, We will discuss a common design issue that causes such incidents and solution for it.

OIC Threading Model:

While designing an integration we need to keep few basic things in mind.

• There are worker threads to process various requests for non-scheduled flows.

• There are scheduler threads to process scheduled flow

• These threads are limited in numbers. Thread count is internal to Oracle and may vary based on instance/subscription type. However we just need to remember that Scheduler thread counts are far lesser than worker thread count.

Design Problem:

In A-team we  have seen large OIC integrations heavily dependent upon scheduled flows, and each flow taking considerable time for execution. For easy understanding let's take a sample scenario:

• In my OIC instance I created 9 different scheduled integrations. Each one has average execution time of 5 minutes.

• All integrations are triggered almost same time say between 7:00 to 7:01 am. When checked on monitoring console there were only 4 instances in running state. What happened to remaining 5 integrations?

•  So I assumed that my OIC instance has only 4 scheduler threads. Therefore, only 4 scheduled flows will run in parallel and threads will be locked until job is complete. And other 5 flows will be waiting until threads are available. Below is the representation of thread assignments (tentative representation)

  • 

• So out of 9, my 4 schedulers executed on time, other 4 are delayed by approx 5 mins and 1 is delayed by 10 mins.

• Now assume if my scheduled integration count in 50 ? Each one executes at frequency of 15-20 mins with average execution time of 3-4 mins. This may create a significant backlog. And If this backlog keeps increasing, You may observe that schedulers are getting delayed by few hours. (e.g : We have seen scenario where a flow scheduled to execute at 11 am was actually executed at 1 pm).

• In real world, You may have different usecases like long running flows scheduled to run once in a day Or short running flows scheduled to run every 20-30 mins. In either case, What we need to remember is if at one point of time there are more scheduled flows than available scheduler threads, then there will be few jobs in waiting state causing delay that could be anywhere between few seconds to few hours.

• In above illustration, Worker threads which are higher in number compare to scheduler threads, are almost idle. So OIC instance is not used efficiently.

Solution:

We can decouple scheduler and integration logic into 2 different flows and let scheduled integration just invoke the child integration using Fire & Forget pattern.

Let's see how we can fix above sample scenario:

• For decoupled design, I created 18 integration flow where 9 are parent (Scheduled integrations) and other 9 are Child (non-scheduled) application integration flows, simply exposed as REST endpoint.

• Child flow exposes REST endpoint with "POST" verb and not returning any response hence making it a "One Way" flow. So it can be invoked as Fire & Forget .

• Each Parent flow invokes its respective child flow. However since child is "one way" flow its not returning anything. So parent flow won't wait for the response after invoking child flow. It will either terminate and release the thread or move to next activity (if any).

• Child integrations will use worker thread for processing.

• So this time, when all 9 scheduled integrations triggered at the same time, they were terminated within few seconds and there were 9 child integrations in "running" state, Running in parallel.

• Within few seconds scheduler threads were idle, ready to execute more scheduled flows.

• Also note that we really don't need to create 9 different scheduled flows. If all 9 child integrations are supposed to trigger around same time then we can build just one parent scheduled integration, invoking each child flow in a sequence. Since these are fire & forget invocations parent flow will just trigger a flow and move to next one without waiting for any response.

• Another benefit of this pattern is, All child integrations are independent flows exposed as REST endpoint. We can invoke it from any other integration or application or just manually, whenever needed. So it can be made as re-usable component.

Integration Flow Snapshots:

Parent Flow:

Child Flow:

Note:

• Invocation of an integration flow from another flow (as parent to child invocation in our scenario) can be made more efficient by enabling feature flag oic.ics.console.integration.invoke.local.integration as described in blog.

• Above feature flag is in "feature controlled" status and not GA at the time of writing this blog. So, There is a risk that it may not become GA and may get rolled-back. So decision to use this feature flag should be taken cautiously.  To understand feature flag lifecycle please refer blog.

Conclusion:

You should decouple your business logic from scheduled integration. Use scheduled integration only to trigger the child integration that runs actual business logic. The invocation from parent flow need not be REST with POST, it could be one-way SOAP as well, but REST is a preferred approach. In either case it has to be one-way.

Best practices to use OIC scheduler and this pattern:

• Never make synchronous call to long running child flows using this pattern, as it would block the parent flow

• Never overuse this pattern specifically for long running batches, since scheduled threads are meant to be long running and not the worker threads.

• Use schedulers to solve batch usecases and not near-realtime one

• There could be usecase where parallelism is not the requirement, scheduled jobs must be invoked in a sequence and users want to trigger dependent jobs with minimal delay. In such cases Parent scheduler flow can contain the business logic (or "synchronously" invoke child flow that contains business logic) and end of the flow can do "Submit-now" to trigger another  scheduled flow. This is a complimentary pattern in-case customers do not need multiple child invokes but want to  trigger another schedule integration after a particular schedule is done.