Monitoring Asynchronous Web Services in Fusion Applications

Introduction

This is part 3 of 4 posts series on Asynchronous Web Services in Fusion Applications. As described in the previous post, Fusion Applications creates a pair of internal, dedicated request and response queues for each domain to support asynchronous web services. This mechanism, of using a request/response pair of queues for all web services within a domain, might raise the following performance concerns: the single queue and its supporting components may potentially become a performance bottleneck under heavy load as the one pair of queues is serving the entire domain and one MDB is serving all web services within an EAR. To address these concerns, we need to monitor Fusion Applications closely and conduct the relevant performance tuning.

Main Article

Fusion Applications control (Enterprise Manager) provides a rich and 360 degree capabilities of overall monitoring that allows users to monitor the status of the infrastructure, domains, servers, deployed applications and their associated performance metrics. See here for details. We recommend always using Fusion Applications control as the starting point to monitor and tune Fusion Applications’ performance.

Monitoring and performance tuning are very big topics to cover. For instance, if a slowdown is observed on asynchronous web services in Fusion Applications, it could have various reasons: network, storage, database, JVM, Weblogic, applications, etc. Generally speaking, we will need to monitor holistically to locate the performance bottleneck and conduct performance tuning.

This post, as the continuation of the previous post which elaborated the internal mechanism of Fusion Applications’ implementation to support asynchronous web services, will drill down and only focus on monitoring those elaborated software components.

AQ Queues

Advanced Queues are the key component that Fusion Applications uses to implement the mechanism of asynchronous web services. The queue length is naturally the key performance indicator that reflects directly if Fusion Applications Asynchronous Web Services work properly and deliver the expected performance. As a best practice, we recommend monitoring the queue depth as the first step of this performance monitoring exercise.

Oracle database defines a view for every single AQ queue. It is very straightforward to query against the view to obtain the queue length:

  • Connect to FA database via your preferred tools such as sqldeveloper or directly via sqlplus with user=FUSION_AQ
  • select count(1) from AQ$CRM_AsyncWS_Request

The output of the above query will provide the queue length showing the number of undelivered messages in the queue.

The table below shows the views defined in Fusion Applications’ database.

AQ Queue View
CRM_AsyncWS_Request AQ$CRM_ASYNCWS_REQUEST
HCM_AsyncWS_Request AQ$HCM_ASYNCWS_REQUEST
FIN_AsyncWS_Request AQ$FIN_ASYNCWS_REQUEST
PRC_AsyncWS_Request AQ$PRC_ASYNCWS_REQUEST
PRJ_AsyncWS_Request AQ$PRJ_ASYNCWS_REQUEST
SCM_AsyncWS_Request AQ$SCM_ASYNCWS_REQUEST
IC_AsyncWS_Request AQ$IC_ASYNCWS_REQUEST
COMMON_AsyncWS_Request AQ$COMMON_ASYNCWS_REQUEST
CRM_AsyncWS_Response AQ$CRM_ASYNCWS_RESPONSE
HCM_AsyncWS_Response AQ$HCM_ASYNCWS_RESPONSE
Fin_AsyncWS_Response AQ$FIN_ASYNCWS_RESPONSE
PRC_AsyncWS_Response AQ$PRC_ASYNCWS_RESPONSE
PRJ_AsyncWS_Response AQ$PRJ_ASYNCWS_RESPONSE
SCM_AsyncWS_Response AQ$SCM_ASYNCWS_RESPONSE
IC_AsyncWS_Response AQ$IC_ASYNCWS_RESPONSE
COMMON_AsyncWS_Response AQ$COMMON_ASYNCWS_RESPONSE

 

An alternative way to monitor AQ queues is via the Enterprise Manager (dbconsole) if it is enabled.

If a significant number of messages are piling up in the queue, we must be vigilant as it means the system is experiencing slow message consumption – the web services of Fusion Applications are not consuming or processing the messages fast enough. This implies that there is a performance bottleneck somewhere in Fusion Applications that prevents Fusion Applications from delivering the expected performance: the most likely reason is that the system is not tuned completely and properly.

Further monitoring and troubleshooting are required to determine the root cause of any message consumption – monitoring and troubleshooting database, network, storage, JVM, WebLogic, etc. In addition, we will also need to monitor other components involved in the asynchronous web services.

MDB

Message Driven Bean (MDB) is the component which consumes and processes messages from the queues. A MDB may have multiple active instances to support concurrent processing. The number of MDB instances varies depending on the MDB configuration at deployment time and the actual work load at runtime.

For example, the CRM-Request-MDB is deployed with the following configuration:

<pool>
    <initial-beans-in-free-pool>1</initial-beans-in-free-pool>
    <max-beans-in-free-pool>16</max-beans-in-free-pool>
</pool>

and the CRM-Response-MDB as follows:

<pool>
    <initial-beans-in-free-pool>1</initial-beans-in-free-pool>
    <max-beans-in-free-pool>5</max-beans-in-free-pool>
</pool>

After consuming from AQ queues, the request-MDB will invoke web service implementation to process the business logic, while the response-MDB will simply make a call back to the client to deliver the response. It makes perfect sense to configure more instances for the request-MDB (16 maximum) than that of the response-MDB (5 maximum).

In the above configurations, the MDB will have one instance initially when it gets deployed and will potentially increase to maximum 16 or 5 instances respectively if WebLogic determines more instances are needed to support higher concurrency.

However, please bear in mind, the concurrency does not rely uniquely on the MDB configuration, it also depends on the thread management mode. Fusion Applications uses WebLogic’s self tuning thread pool with the default work manager, the number of threads allocated for a MDB will vary and follow this pattern: Min(max-beans-in-free-pool,16).

Based on that rule, we can safely conclude that the maximum concurrency of the request-MDB is 16 while the maximum of the response-MDB instances is 5.

The MDB is deployed along with the asynchronous web services implementation(s) within an EAR file. We need to go to the deployed EAR via WebLogic console to do further monitoring. The steps of monitoring MDB are described as follows:

  • Connect to WebLogic console of the domain (CRM for example)
  • Expand Deployments->the EAR containing the MDB in question and web services being invoked
  • Click in the MDB to be monitored

  • Select Monitoring->Running

You might need to click “Customize this table” link to select the information you are interested in and get it displayed on the page.

The value of “Beans In Use Count” provides a count of the number of bean instances currently being used. It is one of the key indicators for the analysis and further investigation. If the value of the request-MDB is equal to or close to 16 and there are messages piling up in the request queue, there might be two possibilities:

  • The web service implementations do not process fast enough, or
  • The current configuration of threads and MDB is not sufficient to support the current load although this is very unlikely in most of situations.

To determine the real cause, close monitoring and troubleshooting of the web service implementations is required. As Fusion Applications implements web services on top of Enterprise JavaBean (EJB), we can monitor the EJBs in the same way as we do for the MDB.

If the value of “Beans In Use Count” is relatively low and we still observe messages piling up in the queue, we will need to monitor and troubleshoot threads, JDBC, database and AQ which possibly prevent the MDB from consuming the messages fast enough from AQ.

Actually, the access to the MDB monitoring page in the console requires us to expand the individual deployed EAR and to get into the monitoring page of the specific MDB. It is very tedious and inefficient to monitor all MDBs in a single, unified view. To address this requirement, it would be nice to write a WLST script which can take a “snapshot” of all MDB to retrieve the statistics of all MDBs in the domain. This could be accomplished through a tool exporting MBean data on a regular basis. You can find an example here from Stefan Koser.

JDBC

Fusion Applications uses a dedicated JDBC datasource to access AQ in the database for asynchronous web services: JRFWSAsyncDSAQ. This datasource is enabled on all servers hosting asynchronous web services EARs.

The JDBC connection pool is configured as follows:

Initial Capacity:0
Min capacity: 0
Max capacity: 100

The datasource can be monitored via WebLogic console as follows:

You might need to click “Customize this table” link to select the information you are interested in and get them displayed on the page.

Threads

Fusion Applications uses WebLogic’s self tuning thread pool with the default work manager. As no custom work manager is currently configured for any kind of components or applications, it means all types of work will be executed with the same priority; the threads to execute this work is allocated and managed by the global default work manager. See the WebLogic documentation for details. We can monitor the overall thread utilization across all types of work in a server.

However, as a result of using the default work manager, it is very difficult to monitor asynchronous web services and its associated components of an individual EAR. Since the number of allocated threads at the same time for an individual MDB is equal to the “Beans In Use Count” described in MDB monitoring, the best way to determine the number of threads associated with a MDB is rather through monitoring the “Beans In Use Count” of the relevant MDB.

In summary, AQ queue length, MDB active instances and JDBC pool for AQ are key indicators of monitoring asynchronous web services. Other monitoring options such as network, database, storage, JVM, threads are also needed alongside. This post, as one of the series on the topics of asynchronous web services, is focused on how to monitor the components relating to asynchronous web service in Fusion Applications. The performance tuning relating to the above monitoring will be introduced in the subsequent post.

Add Your Comment