Where did my MDB concurrency go?

Problem Description

Recently, there have been numerous customers reporting a similar symptom with regards to MDB (Message Driven Bean) concurrency within WebLogic Server.  The complaint is that not all JMS messages are being consumed in a timely manner and in fact some messages look to be consumed in a serial fashion instead of concurrently.

Here is the problem description as reported by a customer:

The producer is publishing 18 messages and out of these 2 are long running messages and 16 are short running. The long running messages sometimes runs more than 30 minutes and the short will finish within 2 minutes. Once the short messages are completed their MDB should be available to pick up the next message. But instead of using a free MDB instance, the message ends up waiting until the long running MDB finishes. Our research seems to indicate that this is an issue with WebLogic where it does not deliver the message to the free consumer and instead delivers it to the “busy” consumer once it finishes processing the long running message.

Problem Analysis

  • With Weblogic Server, the default number of MDB instances that can run concurrently is 16.
  • Let’s say MDB instance 1 and MDB instance 2 are processing the messages that are taking 30 minutes.
  • That leaves us 14 MDB instances to process the remaining 16 messages.
  • MDB instances 3-16 are processing messages 3-16.  That leaves us 2 messages that still need to be processed.
  • After 2 minutes, MDB instances 3-16 are done processing their messages and are now free to pickup the remaining 2 messages.
  • Unfortunately, that is not happening.  Only after MDB instance 1 and/or 2 are done processing their message do the remaining 2 messages get picked up.
  • Instead of the short processing messages being completed after 4 minutes, it is at least 30 minutes before they complete.

Test Case

Let’s use a simplified test case to reproduce what the customer is claiming.

We will use a producer that will send messages with a JMS Property that defines how long the MDB will sleep.
We will use an MDB that will sleep the specified time in the JMS Message’s Property.

Scenario:

  1. 1. MDBs are already running with no messages in the queue they are consuming from.
  2. 2. Submit 1 message where the onMessage takes 60 seconds to complete.
  3. 3. Submit 16 messages where onMessage takes 25 seconds to complete.

With this scenario, since the MDB concurrency (number of MDB instances) is 16, one would expect the total time to process all messages to be 1 minute.

Expected Results (Total Time: 60 seconds):

  1. 1. MDB instance 1 picks up message number 1 and is processing in onMessage. (Time 0)
  2. 2. MDB instance 2 – 16 picks up the next 15 messages and is processing in onMessage. (Time 0)
  3. 3. MDB instance 2 – 16 finish their onMessage processing after 25 seconds. (Time + 25 seconds)
  4. 4. MDB instance 2 (or any of the 15 idle instances) picks up the last message and is processing in onMessage. (Time + 25 seconds)
  5. 5. MDB instance 2 finishes its onMessage processing after 25 seconds. (Time + 50 seconds)
  6. 6. MDB instance 1 finishes its onMessage processing after 60 seconds. (Time + 60 seconds).

Actual Results (Total Time: 85 seconds):

  1. 1. MDB instance 1 picks up message number 1 and is processing in onMessage. (Time 0)
  2. 2. MDB instance 2 – 16 picks up the next 15 messages and is processing in onMessage.(Time 0)
  3. 3. MDB instance 2 – 16 finish their onMessage processing after 25 seconds. (Time +25 seconds)
  4. 4. MDB instance 1 finishes its onMessage processing after 60 seconds. (Time + 60 seconds).
  5. 5. MDB instance 1 picks up the last message and is processing in onMessage. (Time + 60 seconds)
  6. 6. MDB instance 1 finishes its onMessage processing after 25 seconds (Time + 85 seconds)

What’s the deal?

The results from the simplified test case confirm the customer’s complaint: MDB concurrency is not working as effectively as it should be and in fact it feels as though some messages are processed in a serial fashion even though we have idle MDB instances. Before anybody jumps to any conclusions let me quickly state that this is behaving exactly as it has been designed!
In fact, this behavior is an optimization!

Say what? Why would WebLogic have an optimization that causes worse performance? You’re probably thinking that this Eric Gross guy must have taken one too many hits to the head playing hockey. And while that may be true, allow me to explain what is going on.

WebLogic JMS, by default, attempts to be proactive by pushing JMS messages into a JMS Session’s pipeline/backlog. The main reason for doing so is to reduce the amount of time a consumer has to wait for a message to arrive. Without this optimization, messages are only pushed to the JMS Session once the prior message has been acknowledged. So, while a JMS consumer is processing a message in onMessage, why not have WebLogic JMS push more messages to the JMS Session at the same time? That way subsequent messages are immediately available to be processed. Of course, this is very much configurable by a property on the Connection Factory. Weblogic JMS defines this property as “Messages Maximum Per Session” and the default value is 10. For more information on this property, I will refer you to a great blog here: WebLogic JMS Performance Tuning Series, Part 3: Consumer-Side Message Pipelining

Now, with the knowledge about the Messages Maximum Per Session optimization, let’s revisit how it affects this scenario. After all 16 MDB instances are concurrently processing their respective message, the 17th message is pushed to the first MDB instance’s pipeline. If you recall, this first MDB instance is taking 60 seconds to process the first message. Since the 17th message is in the this MDB’s pipeline, it will only get processed AFTER the onMessage completes (after 60 seconds).

Resolution

While 99% of the time the default “Messages Maximum Per Session” optimization improves performance, there are times, as we have demonstrated above, where it can hurt performance. Luckily, it is very easy to fix. All we need to do is change the Messages Maximum Per Session value to 1 (which essentially disables the pipeline) on the custom connection factory here.
Unfortunately, if you are using the default connection factories, the value can not be changed.

Additional Thoughts

You may have noticed that the above scenario deals with MDBs being started BEFORE there are any messages in the queue that they are connected to. So, what happens if there are already messages in the queue before the MDBs are started? In fact, the problem may appear to be even worse because affinity to the client will be maintained until the pipeline is full. The example our documentation mentions outlines this perfectly:

For example: If MessagesMaximum has a value of 10,000,000, the first consumer client to connect will get all messages that have already arrived at the destination. This condition leaves other consumers without any messages and creates an unnecessary backlog of messages in the first consumer that may cause the system to run out of memory.

 

Thanks for reading!

Comments

  1. Raja Mukherjee says:

    Nice article

Add Your Comment