Inside Fusion Middleware 12c: Increasing Scalability with JMS Adapter 12c

JMS Adapter (also known as Oracle JCA Adapter for JMS) is a component available with Oracle SOA Suite or Oracle Service Bus (OSB) which provides a very powerful way to use  the Java Messaging Service (JMS) for sending or receiving messages.

The most important goals for optimizing an SOA Suite or OSB environment are to make sure that

  • Each node of the cluster makes effective use of Java threads and other resources.
  • The cluster will be able to scale efficiently when adding new nodes.

In this article, we will show how JMS Adapter in the new release 12c can be configured in high throughput scenarios to use much fewer threads that in earlier versions. This means, that adding new nodes to the cluster will not require additional threads in all other nodes. As a result, a cluster with many nodes will perform and scale much better.

There are 2 main use cases of JMS Adapter with SOA Suite or OSB:

  • Inbound activation of a composite by receiving messages from a JMS destination (queue or topic)
  • Publishing messages from a composite to a JMS destination

We will focus in this article on the first use case for receiving messages. Only in that scenario – for listening to a queue, the activation framework of the Java EE Connector Architecture (JCA) layer will start a number of threads for JMS Adapter.

First, we will describe how many threads will be created by default or in versions prior to 12c, and then how this changes when using the new feature available with 12c.

Default thread creation for JMS Adapter

For the first example, we will assume a scenario with a 2-node SOA cluster where a Uniform Distributed Queue (UDQ) “jms/TestQueue” is defined and one SOA composite with an inbound JMS Adapter is listening to this queue. See line 2 of the JMS destination overview in Weblogic server in the following table:

jms_queue

With the default behavior, the number of threads used to read messages from this queue in each Java Virtual Machine (JVM) is one. This is derived from the default of the corresponding configuration parameter:

adapter.jms.receive.threads = 1

JMS Adapter creates this specified number of threads for each node in the cluster  to listen to incoming messages on all nodes, so the overall number of threads in each JVM in this example will be two, as shown in the following picture.

2-node-1-thread

We can verify that by taking a thread dump. The thread dump of soa_server1 shows 2 waiting threads like this:

  • “DaemonWorkThread: ‘1’ of WorkManager: ‘default_Adapters'” Id=253 WAITING
  • “DaemonWorkThread: ‘0’ of WorkManager: ‘default_Adapters'” Id=228 WAITING

In the next step, we examine how this picture will change when messages need to be dequeued in parallel. This can be required, for example, if a very high throughput needs to be achieved or if each dequeued message starts a transaction which takes a longer time. Parallel processing can be enabled by increasing the listening threads of each JMS Adapter instance. The property “adapter.jms.receive.threads” can be changed in EM console:

jms adapter properties

After changing this property to 10, the thread count in each thread dump will increase to 20 on each node (40 all together):

2-node-with-10-threads

To show the exponential effect, we will examine the result in a larger SOA cluster. In a 10-node cluster with 10 receive threads, the number of threads will increase to 100 needed on each node and 1000 all together in the cluster (lines in the picture are accurate for the first 3 nodes only):

10-node-unoptimized

The huge negative impact is that adding a new node will increase the number of threads in all other nodes as well. This leads to a non-linear increase of the total threads needed. In a cluster scenario with a lot of queues, this can result in a very high overall number of threads in a single JVM, causing too many context switches of the CPU and slowing down the performance significantly.

JMS Adapter threads in SOA Suite 12c

The new 12c feature to control the thread creation is provided as a new configuration property:

adapter.jms.DistributedDestinationConnectionEveryMember=false  (default is true)

The documentation describes the usage of this property (see chapter 8.3.1.6 in  “Understanding Technology Adapters”):

When true, the JMS Adapter creates a consumer/subscriber for each member of the Distributed Destinations (the author: this is the default and was the setting in 11g). If set to false, the JMS Adapter creates a consumer/subscriber for only local members of the distributed destination.
When the JMS Adapter is connecting to distributed destination on a remote cluster or a server on remote domain, the property ‘adapter.jms.DistributedDestinationConnectionEveryMember’ should always be set to true. When the JMS Adapter is connecting to distributed destination on a local cluster, the property can be set to either true or false. If set to true, the JMS Adapter behavior remains the same as before (that is, there is a consumer for each Distributed Destination is created). If set to false, the JMS Adapter only creates consumer/subscriber for the local members.

We recommend setting this property to false for distributed destinations on a local cluster.

As written in the JMS Adapter documentation (see Appendix), this change is only possible in the composite.xml, not through the Enterprise Manager (EM) web console:

<service name=”JMSConsumer” ui:wsdlLocation=”WSDLs/JMSConsumer.wsdl”>
<interface.wsdl interface=”http://xmlns.oracle.com/pcbpel/adapter/jms/JMSConsumerApp/JMSConsumer/JMSConsumer#wsdl.interface(Consume_Message_ptt)”/>
<binding.jca config=”Adapters/JMSConsumer_jms.jca”>
<property name=”useRejectedMessageRecovery” type=”xs:string” many=”false” override=”may”>true</property>
<property name=”adapter.jms.receive.threads” type=”xs:string” many=”false” override=”may”>10</property>
<property name=”adapter.jms.DistributedDestinationConnectionEveryMember” type=”xs:string” many=”false” override=”may”>false</property>
</binding.jca>

</service>

After changing this property to false, the number of threads used for JMS Adapter in each JVM will drop to 10. All 10 threads on node 1 are listening only to the local UDQ member (on the same node), and so on. The architecture will be much more scalable in a large cluster, as shown in the following picture:

10-node-optimized

It is important to notice that this property cannot be change via Enterprise Manager Console (EM) in 12.1.3 – and it is also not visible in the EM console, even after setting or changing the value in composite.xml.

Receiving messages just from a local queue will in most cases not impact the functional behavior. By using the Weblogic JMS features of message forwarding and load balancing, you can still achieve that all messages are distributed to cluster members with active consumers – without the need to connect to every destination member with JMS Adapter.

Conclusion

In summary, using this property “DistributedDestinationConnectionEveryMember” enables a user to considerably reduce the number of threads created for inbound JMS Adapter instances in a cluster environment. The larger a cluster is, the higher the potential reduction of the number of threads is.

In our previous example of a 10-node cluster with 10 inbound JMS Adapter receive threads, the thread count was reduced by a factor of 10.

This will considerably increase the scalability of the architecture and the performance of each JVM, especially in large clusters with high throughput requirements.

Appendix

References:

Comments

  1. Khatri Rajesh says:

    Hello.

    We are having issues in Production where suddenly Database polling stops working. Also, in some cases ftp file transfer is failing. We see TIMED_WAITING events in the log files that are still analyzing to find the root cause but one thing we observed that default_adapters work manager is not configured. The maximum Minimum Threads Constraint and Maximum Threads Constraint: is not configured. Do we need to setup these values for optimal performance.

    “DaemonWorkThread: ’28’ of WorkManager: ‘default_Adapters'” daemon prio=10 tid=0x00007fc6d0653000 nid=0x40ef waiting on condition [0x00007fc705ddc000]

    java.lang.Thread.State: TIMED_WAITING (sleeping)

    at java.lang.Thread.sleep(Native Method)

    at oracle.tip.adapter.file.inbound.PollWork.run(PollWork.java:413)

    at oracle.integration.platform.blocks.executor.WorkManagerExecutor$1.run(WorkManagerExecutor.java:184)

    at weblogic.work.j2ee.J2EEWorkManager$WorkWithListener.run(J2EEWorkManager.java:184)

    at weblogic.work.DaemonWorkThread.run(DaemonWorkThread.java:30)

    “DaemonWorkThread: ’27’ of WorkManager: ‘default_Adapters'” daemon prio=10 tid=0x00007fc6d0651000 nid=0x40ee in Object.wait() [0x00007fc705edd000]

    java.lang.Thread.State: TIMED_WAITING (on object monitor)

    at java.lang.Object.wait(Native Method)

    – waiting on (a oracle.tip.adapter.file.inbound.FilesToProcess)

    at oracle.tip.adapter.file.inbound.FilesToProcess.dequeueToProcess(FilesToProcess.java:110)

    – locked (a oracle.tip.adapter.file.inbound.FilesToProcess)

    at oracle.tip.adapter.file.inbound.ProcessWork.run(ProcessWork.java:152)

    at oracle.integration.platform.blocks.executor.WorkManagerExecutor$1.run(WorkManagerExecutor.java:184)

    at weblogic.work.j2ee.J2EEWorkManager$WorkWithListener.run(J2EEWorkManager.java:184)

    at weblogic.work.DaemonWorkThread.run(DaemonWorkThread.java:30)

    “DaemonWorkThread: ’26’ of WorkManager: ‘default_Adapters'” daemon prio=10 tid=0x00007fc6d064f000 nid=0x40ed in Object.wait() [0x00007fc705fde000]

    java.lang.Thread.State: TIMED_WAITING (on object monitor)

    at java.lang.Object.wait(Native Method)

    – waiting on (a oracle.tip.adapter.file.inbound.FilesToProcess)

    at oracle.tip.adapter.file.inbound.FilesToProcess.dequeueToProcess(FilesToProcess.java:110)

    – locked (a oracle.tip.adapter.file.inbound.FilesToProcess)

    at oracle.tip.adapter.file.inbound.ProcessWork.run(ProcessWork.java:152)

    at oracle.integration.platform.blocks.executor.WorkManagerExecutor$1.run(WorkManagerExecutor.java:184)

    at weblogic.work.j2ee.J2EEWorkManager$WorkWithListener.run(J2EEWorkManager.java:184)

    at weblogic.work.DaemonWorkThread.run(DaemonWorkThread.java:30)

    “DaemonWorkThread: ’25’ of WorkManager: ‘default_Adapters'” daemon prio=10 tid=0x00007fc6d064c800 nid=0x40ec in Object.wait() [0x00007fc7060df000]

    java.lang.Thread.State: TIMED_WAITING (on object monitor)

    at java.lang.Object.wait(Native Method)

    – waiting on (a oracle.tip.adapter.file.inbound.FilesToProcess)

    at oracle.tip.adapter.file.inbound.FilesToProcess.dequeueToProcess(FilesToProcess.java:110)

    – locked (a oracle.tip.adapter.file.inbound.FilesToProcess)

    at oracle.tip.adapter.file.inbound.ProcessWork.run(ProcessWork.java:152)

    at oracle.integration.platform.blocks.executor.WorkManagerExecutor$1.run(WorkManagerExecutor.java:184)

    at weblogic.work.j2ee.J2EEWorkManager$WorkWithListener.run(J2EEWorkManager.java:184)

    at weblogic.work.DaemonWorkThread.run(DaemonWorkThread.java:30)

  2. If using this property, will it impact unit of order use case, where you want to pick one message for a particular unit of order. Is there a possibility that other node will pick the message in parallel if adapter.jms.DistributedDestinationConnectionEveryMember=false

  3. Is there anyway in 11g configuration in order to achieve the same?

  4. Thanks for there explanations. A very good improvement.

Add Your Comment