Event based Streaming Integration for SCM Items Load

Introduction

In this blog we will look into a pattern to connect Oracle Fusion SCM Product Development (Product Data Hub - PDH) to downstream applications in realtime using an event based streaming integration.

Overview

Here is the use case and an overview of the integration solution.
Items are loaded into Fusion SCM Product Development (PDH) using regularly scheduled Item Bulk load batch job. When the batch completes, a bulk item load event is generated which encapsulates the ESS Request ID and status information of the batch job. An Oracle integration (OIC) flow which subscribes to the SCM Item bulk load event is triggered on completion of the batch job. The Integration flow then queries Fusion SCM for list of Items loaded in the batch, iterates through each Item, enriching by additionally querying required Item Info attributes and child structures as needed. The integration then publishes every item info to an OCI stream partition (OSS) for further downstream processing by downstream applications. 

Pattern

The pattern involves a event driven integration that is triggered by an event of interest occuring on Fusion SCM. The processing further handed down to an asynchronous integration to help deal with high volume item load synchronization downstream in a performant manner.

Details

The further section will detail the integration solution. Note that the solution is similar on both OIC Gen3 as well as OIC Gen2 and screenshots from both flavours of OIC are shown to remind of this fact.

Connections

Below are the connections used in the integration

• A Fusion ERP/SCM cloud adapter connection that is used for event subscription as well as to query item details from Fusion SCM
• A Rest trigger connection for asynchronous one way integration to the streams dispatcher integration
• OCI Streams Connection to publish every single item information to OCI streams for further downtstream processing

Integrations

• Bulk Items Load Event Listener
• Streams Dispatcher

Below screenshots show in brief the overall integrations, configurations used , rough mappings between payloads and scope level fault handler
 

Event Listener Flow

Here are some of the salient steps from the above Event listener integrtaion.

Firstly the Trigger which is configured to listen to Item Bulk Load event as shown below

The received event payload consists of the Request ID of the ESS Job that performed the bulk load of items into Fusion SCM. Shown below are the mapping task and invoke operation on SCM connection using findItem operation on ItemService. They are responsible to fetch all the individual Item Numbers from the Bulk Item process

Note that the query of Items is limited to 500 items per fetch to avoid the below Fusion SaaS error when invoking the FindItem Service

For requirements where there are more than 500 items per batch, the below approaches can be adopted 
    1) Break the ESS batches into smaller chunks so that each batch processes less than 500 item load into PDH
    2) Use an outer while loop in BulkItemsLoad EventListener which passes 500 items in every turn to the ForEach Iterator, until there are more items to process.

Next, here is the For-Each iterator that invokes the Stream Dispatcher asynchronously for every item from the ESS batch

Note the scope task within the For-Each iterator. The scope handler allows us to attach a scope level fault handler to handle any errors while processing individual items without interrupting the entire batch

Depicted above is a simple scope handler that logs the fault information and proceeds iterating to other items in the batch. But practically, the scope handler could be used to notify operations about the individual failures so they can be recovered manually after correcting any errors.

Streams Dispatcher flow

The streams dispatcher is a simple asynchronous integration that gets an Item id as input. It fetches the required items information using Items SOAP or REST services. This enriched Items information is then sent as payload content for OCI streaming service OSS. Further downstream, we clould have multiple subcribers that consume the Item details and process them in various end systems. The subscriber flows are not deplicted in this blog.

Product Doc links -    Rest API for Fusion SCM     SOAP Services for Fusion SCM    SCM 22D APIs

Below picture shows a sample mapping to fetch Item details from Fusion SCM using the ItemService 

The retrieved Item details are then mapped to streams payload. Streaming supports various message structure options like Avro, JSON and XML. Shown here is message publish without message structure, where the entire payload is mapped to streaming message's content field.

Downstream processing

Once the enriched Items information are published to OCI streams, further downstream integrations using OCI Streams connection can be used to consume and process them to downstream applications.

Conclusion

We have seen how to build event driven integrations to synchronize business objects from Fusion SCM with external applications asynchronously using OCI Streaming sevice