Tuning Asynchronous Web Services in Fusion Applications Part II

Introduction

In a series of earlier blogs we have covered in detail several aspects of asynchronous web services in Fusion Applications including the general implementation, how to monitor, and also discussed tuning considerations and options. This article discusses an additional tuning option for how Fusion Applications consumes waiting messages in the underlying queues.

Main Article

In a recent engagement tuning a high-volume environment we have discovered a situation where a large Fusion Applications cluster can lead to contention on the database when a high number of application threads concurrently attempt to dequeue messages from the Advanced Queues (AQ) used for the asynchronous web service implementation. We were lucky to have the database experts from the Real World Performance group with us supporting the database tuning.

A detailed analysis revealed that the default MDB implementation (detailed discussion) leverages the AQjmsConsumer.receive operation which can result in database side contention if the number of MDB threads and, hence, the corresponding database sessions were increased in order to achieve higher parallelism on the application tier. AQjmsConsumer.receive is implemented in a way to delegate the active polling for new messages from the queue to the database tier and the database session will repeatedly try to lock a message and deliver this message back to the waiting application tier thread. Now with a significant number of such database sessions technically competing for messages, a high portion of these dequeue attempts can’t be successful as only a single session can be successful in locking a record. In addition to this, the use of logical queues leveraging a message selector on top of a single physical queue (again, details here) causes that an enqueue triggers dequeue attempts (using a messaging mechanism within the database) for all sessions waiting for a message on the same physical queue even if the new message is for a different logical queue. Overall, this default logic can cause relevant overhead on the database tier as observed through a high number of ‘select for update skip locked’ invocations in the usual database performance reports while only a small portion of the invocations actually deliver a record.

In order to address this situation, there is a new patch available for the Weblogic MDB implementation which allows for leveraging the non-blocking operation AQjmsConsumer.receiveNoWait instead of AQjmsConsumer.receive. With that, the control is passed back from the database to the application tier immediately if the the process cannot get hold of a message (i.e. no looping happens on the database tier) and put the application thread to sleep for a defined duration before attempting the next dequeue. This results overall in a significant reduction of database contention and unsuccessful row locking attempts while not compromising the overall throughput.

In order to activate the non-blocking dequeue logic, the corresponding patch 21561271 delivered through P4FA patch bundles must be available on the system. Secondly, there is a new Java system property in order to turn on the new logic by setting weblogic.ejb.container.AQMDBReceiveNoWait=true. Per default this switch is set to false for backward compatibility reasons, i.e. it needs to be explicitly turned on to use the non-blocking implementation.

Finally, in this analysis there was a parallel effort in order to optimise the database side implementation as well resulting in patch 16739157 which is recommended to be applied if high number of row lock attempts are observed on Advanced Queues.

Conclusion

This article presented an additional tuning option for decreasing database side contention for deployments with a larger number of application tier processes dequeuing from the same Advanced Queue.

Add Your Comment