Configuring SOACS for BPEL invocations of OAuth2 protected services

OAuth2 has become increasingly popular for authorizing access to web services.  Invoking these services is possible by applying Oracle Web Services Manager (OWSM) policies to the component in the composite.  OWSM actually acquires the OAuth Access Token and includes it in the HTTP request to the resource server for you automatically.  This blog describes the […]

EDI Processing with B2B in hybrid SOA Cloud Cluster integrating On-Premise Endpoints

Executive Overview SOA Cloud Service (SOACS) can be used to support the B2B commerce requirements of many large corporations. This article discusses a common use case of EDI processing with Oracle B2B within SOA Cloud Service in a hybrid cloud architecture. The documents are received and sent from on-premise endpoints using SFTP channels configured using […]

Custom Message Data Encryption of Payload in SOA 11g

Introduction This article explains how to encrypt sensitive data (such as ssn, credit card number, etc ) in the incoming payload and decrypt the data back to clear text (or original form) in the outgoing message. The purpose is to hide the sensitive data in the payload, in the audit trail, console and logs. Main […]

How to Recover Initial Messages (Payload) from SOA Audit for Mediator and BPEL components

Introduction In Fusion Applications, the status of SOA composite instances are either running, completed, faulted or staled. The composite instances become staled immediately (irrespective of current status) when the respective composite is redeployed with the same version. The messages (payload) are stored in SOA audit tables until they are purged. The users can go through Enterprise […]

New BPEL Thread Pool in SOA 11g for Non-Blocking Invoke Activities from 11.1.1.6 (PS5)

Up to release 11.1.1.5 there have been 4 thread pools in Oracle SOA Suite 11g to control parallelism of execution:

  • Invoke Thread Pool (for asynchronous invocations)
  • Engine Thread Pool (i.e. for callback execution)
  • System Thread Pool
  • Audit Thread Pool

Starting with 11.1.1.6 there is one (still undocumented) new thread pool introduced for non-blocking invoke activities.

Here is a view of the System MBean Browser:

image

The MBean name is: 
oracle.dms:Location=soa_server1,name=/soainfra/engines/bpel/requests/non-block-invoke,type=soainfra_bpel_requests

You can change a synchronous invoke activity from a blocking call to non-blocking by using the partnerlink level property:

image

This thread pool is configured in SOA-Administration –> BPEL Service Engine Properties under “More BPEL Configuration Properties…” with the property DispatcherNonBlockInvokeThreads:

image

Be aware that the default is only 2 – so this can be a bottleneck in high load scenarios if not changed. Especially if you have multiple partnerlinks using non-blocking calls – because all of them share this thread pool…

Have fun, Stefan

Retrieving and Setting HTTP Headers in BPEL

The capability to retrieve and set HTTP headers in BPEL was recently added to Oracle SOA Suite 11g. Edwin Biemond has written an excellent blog post on how to use this capability.

From a security/IDM perspective, I think this feature opens up the ability to create some interesting solutions whereby identity information is added to HTTP headers by OAM (or other SSO products) in the web tier and consumed by services in the app tier. It also makes it possible to pass identity data between services in HTTP headers and thereby ignore having to modify web service requests themselves.

I’ll only add as a warning to remember that end users have the capability to add whatever HTTP headers they want to the requests they make. So, solutions should be developed with this in mind. In particular, if you are going to create a solution that depends on BPEL consuming an HTTP header created by an OAM response, you need to take steps to either ensure that this header really came from OAM (by signing or encrypting it) or take steps to ensure that all requests to BPEL really did originate by coming through the web tier with OAM.

Encapsulating OIM API’s in a Web Service for OIM Custom SOA Composites

Introduction This document describes how to encapsulate OIM API calls in a Web Service for use in a custom SOA composite to be included as an approval process in a request template. We always recommend customers to follow this approach when trying to invoke OIM’s APIs inside SOA composites used as approval processes for the […]

Choosing BPMN or BPEL to model your processes

With Dave Shaffer Recently, I had a discussion with several colleagues and one of our large customers about when to use BPMN and when to use BPEL to model business processes.  I have discussed this topic before in this post … Continue reading

How should I generate an external reference to a composite instance?

If you have a requirement to generate an external reference to an instance within SOA suite, how should one go about it? Which one of the values should you use?

Event Delivery Network (EDN) – A practical example

Details of how Oracle Event Delivery Network is used may be found here

A lower level description of EDN can be found here.

List of all states from COMPOSITE_INSTANCE, CUBE_INSTANCE, DLV_MESSAGE tables

In many of my engagements I get asked repeatedly about the states of the composites in 11g and how to decipher them, especially when we are troubleshooting issues around purging. I have compiled a list of all the states from the COMPOSITE_INSTANCE, CUB…

OIM 11g OID (LDAP) Groups Request-Based Provisioning with custom approval – Part II

Introduction This is Part Two of the article describing a potential implementation of Request Based LDAP Group Membership provisioning. Part One can be accessed here. Continuing with the implementation after disabling the default approval policies at the Request and Operation Levels, the next step is to configure OIM to enable the modification of a provisioned […]

OIM 11g OID (LDAP) Groups Request-Based Provisioning with custom approval – Part I

Introduction In recent days, I was assigned the task to implement a use case that I am sure many customers of Oracle have in mind but are not sure how to implement in OIM 11g. I even saw some thread inquiring about this very topic with no answer. Well, after some time I was able […]

How to Skip the Handoff Between BPEL and the Adapter Layer to Improve Performance

Say you have a simple BPEL process that gets messages from JMS queue through a JMSAdapter, transforms the message, and then calls a down stream service via SOAP.

For such a simple BPEL process, are there any optimization that can be done for performance?

The answer is yes. And the low hanging fruits can be picked from layer between the inbound JMSAdapter and BPEL process.

The Problem

Just open up the WSDL file of the inbound adapter. You will notice something like the following:

<portType name="Consume_Message_ptt">
<operation name="Consume_Message">
<input message="tns:ExpenseRecord_msg"/>
</operation>
</portType>


Notice there is only an INPUT tag but no OUTPUT tag? This means that the adapter is sending a one-way message to BPEL.

One-way message is one of the interaction patterns between the caller and the BPEL process that Oracle BPEL supports. When a BPEL process is initiated by an adapter, the adapter acts more or less like a regular SOAP caller to the BPEL process, except calls will go through a more efficient native binding protocol instead of SOAP. As such, the interaction between the two COULD BE any of the interaction patterns above. But when you wire this adapter to BPEL in JDeveloper, JDeveloper by default chooses the “one-way message” interaction pattern. Hence the WSDL file above.

By default, incoming one-way messages are saved in the delivery service database table dlv_message (it is invoke_message table in 10G). These requests are later acquired by Oracle BPEL Server worker threads and delivered to the targeted BPEL process. Such two-phase interaction can be illustrated as follows:



Figure 1. Default Setting of One-Way Interaction (oneWayDeliveryPolicy=async)

Separating message sending and message processing, and saving message to the database make sense for a lot of cases. The biggest benefit is that allows the sender of message to “fire and forget” – swiftly send the message and get done with it, even though the actual processing of the message make take longer time. One-way interaction is like email: with one click of the button you send an email, while it may make take the recipient of the email 1 hour to read and act on it.

The other benefit is message reliability: Even if the SOA server is down before processing the message, the sender rest for sure that the message is reliably saved to the database so that it can still be processed once the server is up again.

But the same benefits may not make sense for an inbound JMSAdapter.

First the adapter probably doesn’t care about fire and forget. What does matter is usually the overall speed or throughput of send+process, instead of just “send”.

Second, the message was already saved to the inbound message queue before being picked up. It does not need to be saved at another place for the sake of reliability.

When the benefits of have separate threads to send and process the message are less meaningful to adapter, its cost becomes more stand-out: it introduces extra latency by saving the message to database. In short, this interaction pattern pays the cost without enough benefits.

The Improvement

If we combine the two threads, and disable the insert into the database table, load against the database is reduced, latency of the insert will be eliminated, and the extra work the container was doing to manage it is skipped. Skipping this layer may lead to improved response times.

It takes two steps to achieve the improvement and these two steps must be done in tandem.

Improvement Step 1: Disabling saving of one-way message to database

In SOA 10.1.3.x, you can set the deliveryPersistPolicy=off.immediate in the bpel.xml of the BPEL process in question.

<BPELSuitcase>
<BPELProcess id="ProcessFulfillmentOrderBillingBRMCommsAddSubProcess" src="ProcessFulfillmentOrderBillingBRMCommsAddSubProcess.bpel">
...
<configurations>
...
<property name="deliveryPersistPolicy" encryption="plaintext">off.immediate</property>
</configurations>
</BPELProcess>
</BPELSuitcase>


In SOA 11G, you should set the bpel.config.oneWayDeliveryPolicy=sync in the composite.xml of the composite application.

...
<component name="myBPELServiceComponent">
....
<property name="bpel.config.oneWayDeliveryPolicy">true</property>
</component>

For details of how the oneWayDeliveryPolicy works, please refer to the 13 Oracle BPEL Process Manager Performance Tuning of the Oracle® Fusion Middleware Performance and Tuning Guide. The 10G equivalent is http://docs.oracle.com/cd/B32110_01/core.1013/b28942/tuning_bpel.htm

For details of how to set this property in 11G, please refer to Appendix C Deployment Descriptor Properties of Oracle® Fusion Middleware Developer’s Guide for Oracle SOA Suite. For 10G, please refer to Oracle® Application Server Performance Guide 10g Release 3 (10.1.3.1.0)

Figure 2. Results of Improvement Step 1 (oneWayDeliveryPolicy=sync)

Improvement Step 2: Modifying the threads to increase concurrency

Now you have done step 1 to skip saving the one-way message to database. But just by looking at the picture above you may already notice there is a problem.

By skipping saving one-way message to database table, the adapter threads become the sole control of concurrency. The famous BPEL Invoke threads and engine threads do not apply anymore. This essentially means the application becomes single threaded, and the throughput may drop.

That is is why setting deliveryPersistPolicy=off.immeidate (in 10G) or oneWayDeliveryPolicy=sync (11G) should be done in tandem with increasing the number of adapter threads. Otherwise, you will likely create a drop in throughput. Optimal values of adapter threads should be determined through performance testing.

Figure 3. Results of Improvement Step 2 (oneWayDeliveryPolicy=sync with Adapter Threads)

For JMSAdapter in 11G, you can set the adapter threads in the the properties for tuning the thread counts of various adapters are listed as follows.

<service name="dequeue" ui:wsdlLocation="dequeue.wsdl">
<interface.wsdl interface="http://xmlns.oracle.com/pcbpel/adapter/jms/textmessageusingqueues/textmessageusingqueues/dequeue%2F#wsdl.interface(Consume_Message_ptt)"/>
<binding.jca config="dequeue_jms.jca">
<property name="adapter.jms.receive.threads" type="xs:string" many="false">10</property>
</binding.jca">
</service>

For 10G BPEL, you set the JMSAdapter receiver threads at bple.xml of BPEL:

<activationAgents>
<activationAgent className=“…" partnerLink="MsgQueuePL">
... <property name="adapter.jms.receive.threads”>5</property>
</activationAgent>
</activationAgents>

For 10G ESB, you set the JMSAdapter receiver threads at *esbsvc file of OESB:

<service name="ListenForNewEmployees" ….>
...
<endpointProperties>
<property name="adapter.jms.receive.threads" value="10"/>
</endpointProperties>
</service>

The ways to configure adapter threads vary between adapters and product versions. Please refer to documentation for details. I will have another post soon on this subject.

More Benefits

After setting the oneWayDeliveryPolicy=sync (or deliveryPersistPolicy=off.immediate for 10G) AND configure multiple threads, you achieve shorter latency and higher throughput.

But besides the gain in performance, you can also enjoy the following side benefits:

1. Ease of throttling

When oneWayDeliveryPolicy=async, two layers control the incoming rate of messages into BPEL service engine: the adapter threads AND the BPEL invoke threads (see Figure 1 above). And because Invoke Threads are shared among BPEL processes running on the same JVM, you can’t just throttle for one process without affecting another.

Now setting adapter threads along with oneWayDeliveryPolicy=sync, you only need to throttle the incoming load with adapter threads, which only affect the SOA component that the adapter calls. You can throttle per adapter individually.

2. Avoid Back Log Messages Flooding the SOA server

If there is an outage of SOA servers and the adapter stops dequeuing from the inbound message queue, the enterprise application on the other side of the queue may continue to enqueue. That builds up a large back log in the inbound message queue.

By the time the SOA servers are restarted, with oneWayDeliveryPolicy=async, the large back log of messages in the queue may flood in the SOA servers, saturating all the Invoke Threads and leave no Invoke Threads to work for other BPEL processes. In contrast, there is no such worry if you throttle the incoming loads to the BPEL process by setting oneWayDeliveryPolicy=sync along with adapter threads.

3. Ease of Monitoring:

Rather than checking both the inbound message queue AND the dlv_message/invoke_message table at database for the flow rate of messages into BPEL service engine, you have one place to monitor: the inbound message queue.

Oracle SOA 10G Tricks: Optimize calling path between ESB and BPEL

In SOA 10G, it is a good practice to have BPEL and ESB co-located at the same JVM. Doing so will not only improves performance but also allows JTA global transaction propagates between the two. However, these advantages will not be fully materialized, …

Invoking web services dynamically in BPEL 1.1

This topic has been covered in postings that can be found at various locations. Your favourite search engine will reveal them to you.

Unfortunately, not all of these posts are up-to-date and some are just confusing. My intention here is to demonstrate a practical example from beginning to end using modern Oracle technologies. I have used JDeveloper 11.1.1.5.0 and Weblogic Server 10.3. I am assuming that the reader is familiar with JDeveloper, deploying to Weblogic and using the Weblogic Enterprise Manager (for testing purposes).

Before one can invoke a web service via its well-known endpoint, it is important to understand a significant restriction of this technique. Each web service must have identical message, port and namespace definitions. These need to be known at design time. So what’s the point of this? The benefit of this technique is that services located at different endpoints may be invoked based on runtime decisions (data driven). Indeed, the payload of the input message to the “driver” code [ see later ] may even contain the endpoint address. Thus, different functionality can be easily achieved by being able to invoke completely different services or perhaps just different versions of a service (providing the different versions are deployed and their endpoints are well known).

The development of this demonstration is strictly “bottom up”. We need to define our web service message structure. Implement and deploy the web service. “Clone” that service as many times as is necessary to give the number of services you need [ more detail later ]. We’ll just build 2 services.

Here’s the schema definition for our messages:-

<?xml version=”1.0″ encoding=”windows-1252″ ?>
<xsd:schema xmlns:xsd=”http://www.w3.org/2001/XMLSchema
            xmlns=”http://www.oracle.com/ateam/dynamic
            targetNamespace=”http://www.oracle.com/ateam/dynamic
            elementFormDefault=”qualified”>
  <xsd:element name=”input”>
    <xsd:complexType>
      <xsd:sequence>
        <xsd:element name=”Endpoint” type=”xsd:string”/>
        <xsd:element name=”parameter” type=”xsd:string”/>
      </xsd:sequence>
    </xsd:complexType>
  </xsd:element>
  <xsd:element name=”output”>
    <xsd:complexType>
      <xsd:sequence>
        <xsd:element name=”parameter” type=”xsd:string”/>
        <xsd:element name=”status” type=”xsd:string”/>
      </xsd:sequence>
    </xsd:complexType>
  </xsd:element>
</xsd:schema>

For the purposes of this example, we have an input message that contains Endpoint (being the endpoint of the service we intend to invoke) and parameter which is an arbitrary value that the service will “echo” to the invoker.

The output contains parameter (being the value that will be echoed) and status which will contain some value unique to the service.

So let’s start to design, build and deploy our project.

Create a new SOA application in JDeveloper and call it ATeamDynamic and specify an Application Package Prefix of your choice.

Name the Project Service_1 and we’ll start with an empty Composite.

Create an XML Schema Definition in the project’s xsd folder and call it common.xsd. Don’t worry about the namespace declaration because we’ll now overwite the xsd stub. Populate the schema with the xml shown above. Also, please note that for the sake of brevity we will use this schema for both the service(s) and the “driver” even though the Endpoint element is only significant in the driver process (i.e. not used by the service itself).

Drag a BPEL process onto the Composite canvas. Ensure that BPEL Spec’ 1.1 is checked. Name the BPEL activity as Process. Specify http://xmlns.oracle.com/ATeamDynamic/Service/Process as its namespace. Select Synchronous BPEL Process as the template. [ The service does not need to be synchronous for it to be usable dynamically but, obviously, the driver would have to be constructed differently if it was asynchronous ] The service name should read process_client.

Expose as a SOAP Service should be checked. Use the element browser to locate the input and output messages from the schema definition we created earlier.

Double-click the BPEL activity to go into the BPEL designer.

Drag an Assign activity onto the BPEL design canvas between the receive and reply activities.

Double-click the assign and do this:-

Assign activity

So all we’re doing is to copy (echo) the parameter value and set the status to “Service 1”.

Deploy the Project to Weblogic. In Enterprise Manager, test this service to ensure that it runs as expected. The tester will require an Endpoint value even though it’s not used. The parameter value should be echoed and the Status should indicate “Service 1”.

We will now create the second service. In JDeveloper, create a new SOA Project and name it Service_2. The procedure is exactly the same as for Service_1 except that we don’t need to create the xsd (we can re-use what we’ve already built) and in the BPEL Assign, we will copy the expression “Service 2” to the status element. Everything else is identical.

Now for the exciting part – the Dynamic Driver.

Create a new SOA Project and call it DynamicDriver and we’ll start with an empty Composite.

Drag a BPEL process onto the canvas. The name and namespace can be of your choosing. Use the Synchronous template for ease of testing. We will expose this as a SOAP service and use the same message types as before – i.e. from the common.xsd we created earlier.

Double-click the BPEL process.

Drag a PartnerLink Web Service Construct into the right-hand swim lane. The name can be anything of your choosing. We now need the deployed WSDL for one of the services we built / deployed earlier (it doesn’t matter which one). So, in the Create PartnerLink Wizard, we need to enter the WSDL URL. The easiest way to do this is to go back to the Enterprise Manager test page for Service_1 (or Service_2). On the test page you will see the WSDL query URL. It will look something like this:-

http://<hostname>:7001/soa-infra/services/default/Service_1/process_client_ep?WSDL

Copy and paste this URL into the Wizard. Click on PartnerLink Type. It will show as Process. Click on PartnerRole and select ProcessProvider. Click OK – i.e. leave My Role as “–Not Specified–“.

Drag an Invoke Construct onto the canvas between the receive and reply activities.. Connect the invoke to the PartnerLink. Create input and output variables as input and output respectively. [ I chose short generic names for brevity ]

We now need a new schema definition. We’ll start by creating a stub. Create a new schema definition in the xsd folder of the DynamicDriver project. Name it ws-addressing.xsd (although it could be called anything). Don’t worry about its content yet.

In your favourite browser, enter this URL http://schemas.xmlsoap.org/ws/2003/03/addressing/

This will reveal the ws-addressing schema that you need. Use your browser’s “view source” mechanism then copy and paste into your newly created ws-addressing.xsd

Now let’s go back to the BPEL Editor and we’ll create a new variable. Call it EndpointReference. Click the radio button labelled Element then browse for the EndpointReference element in the ws-addressing schema.

Drag an Assign construct onto the canvas immediately before the Invoke.

We now have to implement three copy activities within the Assign.

Firstly, copy this XML fragment into the EndpointReference variable:-

<EndpointReference xmlns=”http://schemas.xmlsoap.org/ws/2003/03/addressing” xmlns:ns1=”http://xmlns.oracle.com/AteamDynamic/Service/Process”><Address/><ServiceName>ns1:process_client_ep</ServiceName></EndpointReference>

Note that the ns1 namespace declaration and the ServiceName depend entirely on the developer having followed the naming conventions described herein.

We then need to copy the Endpoint part of the inputVariable to the Address part of the EndpointReference.

The third and final step for setting up the PartnerLink is simply to copy the EndpointReference to the PartnerLink. It should look like this:-

Endpoint assignment

Unusually, this may be easier to read in source mode thus:-

[ Please note that at deployment (build) time, a warning will be emitted concerning the 2nd copy because of a type mismatch. This is expected and may be ignored ]

Now we need to populate the payload for the Web Service itself thus:-

Assign5

The WS doesn’t use the Endpoint value but we need to copy something as the schema dictates that this element is mandatory.

After the Invoke, we’ll need to get the WS response like this:-

Assign6

Now we can deploy the DynamicDriver Project and test it.

In Enterprise Manager, start by selecting the Service_1 Project and then the Test tab. There you will see the endpoint URL for this service. Copy this URL.

Now select the DynamicDriver Project then the Test tab. Paste the Service_1 URL into the Endpoint field followed by some arbitrary text in the Parameter field. Click Test Web Service. If all is well, you will see the parameter value echoed and a status of “Service 1” indicating that the correct WS has been called.

Repeat the test for Service_2.

The complete Application is available here

Purging old instance data from SOA/BPEL 10g

My colleague Deepak Arora (see his blog here) has written an excellent white paper on purging old instance data from SOA Suite/BPEL 10g and also a great blog post on automating the deletion of old partitions.  If you are interested in … Continue reading

Human Workflow in 11g

Update:  If you are interested in Human Workflow, you may also find this post interesting.  It’s not about workflow per se, but it does show how to use a simple workflow in a BPMN process on 11g. I am often … Continue reading

Calling BPEL web services from Application Express

Just a quick post, on an issue that seems to come up a lot – how to call a BPEL web service from Application Express.  This used to be quite challenging, but now, with APEX 3.1.2, APEX can understand a … Continue reading