Recently, there have been numerous customers reporting a similar symptom with regards to MDB (Message Driven Bean) concurrency within WebLogic Server. The complaint is that not all JMS messages are being consumed in a timely manner and in fact some messages look to be consumed in a serial fashion instead of concurrently.
Here is the problem description as reported by a customer:
Let's use a simplified test case to reproduce what the customer is claiming.
We will use a producer that will send messages with a JMS Property that defines how long the MDB will sleep.
We will use an MDB that will sleep the specified time in the JMS Message's Property.
With this scenario, since the MDB concurrency (number of MDB instances) is 16, one would expect the total time to process all messages to be 1 minute.
The results from the simplified test case confirm the customer's complaint: MDB concurrency is not working as effectively as it should be and in fact it feels as though some messages are processed in a serial fashion even though we have idle MDB instances. Before anybody jumps to any conclusions let me quickly state that this is behaving exactly as it has been designed!
In fact, this behavior is an optimization!
Say what? Why would WebLogic have an optimization that causes worse performance? You're probably thinking that this Eric Gross guy must have taken one too many hits to the head playing hockey. And while that may be true, allow me to explain what is going on.
WebLogic JMS, by default, attempts to be proactive by pushing JMS messages into a JMS Session's pipeline/backlog. The main reason for doing so is to reduce the amount of time a consumer has to wait for a message to arrive. Without this optimization, messages are only pushed to the JMS Session once the prior message has been acknowledged. So, while a JMS consumer is processing a message in onMessage, why not have WebLogic JMS push more messages to the JMS Session at the same time? That way subsequent messages are immediately available to be processed. Of course, this is very much configurable by a property on the Connection Factory. Weblogic JMS defines this property as "Messages Maximum Per Session" and the default value is 10. For more information on this property, I will refer you to a great blog here: WebLogic JMS Performance Tuning Series, Part 3: Consumer-Side Message Pipelining
Now, with the knowledge about the Messages Maximum Per Session optimization, let's revisit how it affects this scenario. After all 16 MDB instances are concurrently processing their respective message, the 17th message is pushed to the first MDB instance's pipeline. If you recall, this first MDB instance is taking 60 seconds to process the first message. Since the 17th message is in the this MDB's pipeline, it will only get processed AFTER the onMessage completes (after 60 seconds).
While 99% of the time the default "Messages Maximum Per Session" optimization improves performance, there are times, as we have demonstrated above, where it can hurt performance. Luckily, it is very easy to fix. All we need to do is change the Messages Maximum Per Session value to 1 (which essentially disables the pipeline) on the custom connection factory here.
Unfortunately, if you are using the default connection factories, the value can not be changed.
You may have noticed that the above scenario deals with MDBs being started BEFORE there are any messages in the queue that they are connected to. So, what happens if there are already messages in the queue before the MDBs are started? In fact, the problem may appear to be even worse because affinity to the client will be maintained until the pipeline is full. The example our documentation mentions outlines this perfectly:
For example: If MessagesMaximum has a value of 10,000,000, the first consumer client to connect will get all messages that have already arrived at the destination. This condition leaves other consumers without any messages and creates an unnecessary backlog of messages in the first consumer that may cause the system to run out of memory.
Thanks for reading!