Tuning Asynchronous Web Services in Fusion Applications

This post is the 4th part (out of 5) of the series of Fusion Applications asynchronous Web Services. In the previous posts, we have elaborated:

This post will focus on how to conduct performance tuning of the asynchronous web services in relation to Fusion Applications. As mentioned in the previous post, performance tuning is a very broad topic which covers many aspects such as Operating System, network, storage, database, JVM, WebLogic, etc. In this post, I will focus only on the areas which are specific to the components supporting the asynchronous web services in Fusion Applications.

Tuning AQ Queues

Oracle Advanced Queuing (AQ) is the foundation of asynchronous Web Services in Fusion Applications. Based on the monitoring introduced in the previous post, if you observed messages piling up in the AQ queues, we might need to troubleshoot and conduct performance tuning on the AQ queues and the database.

The following two documents, which can be found in the Oracle knowledge base within My Oracle Support,provide a very handy and useful set of information about how to troubleshoot the AQ queues and the recommendations of tuning practices.

  • Doc ID 335516.1 – Master Note for Streams Performance Recommendations for the detailed tuning practice.
  • Doc ID 730036.1 – Troubleshooting Oracle Streams Performance Issues.

Tuning MDB Concurrency

The current Message Driven Bean (MDB) concurrency configured in Fusion Applications should work properly in most situations. If there is no specific reason, you shouldn’t have the need to change any settings unless you are certain that the MDB concurrency is identified as the performance bottleneck.

In the case where the message arrival rate is high and constant or the Web Service implementation constantly takes a long time to execute, for example, due to complex business logic, you might want to increase the MDB concurrency. If this is the case, you can tune the MDB concurrency by dealing with both the thread management and the MDB instances.

The MDB configuration is defined in the EJB deployment descriptor. The following steps illustrate how to access a MDB’s deployment descriptor of a Fusion Applications EAR, and how to make the relevant changes.

  • Login to WebLogic Console of the domain to be tuned. For example, CRM domain.
  • Click Deployments
  • Click the EAR in which the MDB needs to be tuned, for example, CustomerApp(V2.0)

wls-console-deployment

  • The location of the EAR is showed by Path: /u01/app/fa/fusionapps/applications/crm/deploy/EarCustomer.ear
  • Use a SSH session to access the above path: cd /u01/app/fa/fusionapps/applications/crm/deploy/EarCustomer.ear
  • You will see all the Customer EAR’s artifacts are located under this directory
  • The MDB’s information is packaged in AppAsyncMdb.jar
  • Copy AppAsyncMdb.jar to a temp directory
  • Change to the temp directory
  • Extract the AppAsyncMdb.jar : jar -xvf AppAsyncMdb.jar
  • Open the file META-INF/weblogic-ejb-jar.xml
<weblogic-ejb-jar xmlns="http://www.bea.com/ns/weblogic/10.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.bea.com/ns/weblogic/10.0 http://www.bea.com/ns/weblogic/10.0/weblogic-ejb-jar.xsd">
  <weblogic-enterprise-bean>
    <ejb-name>ZcmMigrationService_AsyncRequestProcessorMDB</ejb-name>
    <message-driven-descriptor>
      <pool>
        <initial-beans-in-free-pool>1</initial-beans-in-free-pool>
        <max-beans-in-free-pool>16</max-beans-in-free-pool>
      </pool>
      <destination-jndi-name>oracle.j2ee.ws.server.async.AQRequestQueue</destination-jndi-name>
      <connection-factory-jndi-name>aqjms/AsyncWSAQConnectionFactory</connection-factory-jndi-name>
    </message-driven-descriptor>
    <jndi-name>ZcmMigrationService_AsyncRequestProcessorMDB</jndi-name>
    <run-as-principal-name>FUSION_APPS_CRM_ASYNC_WS_APPID</run-as-principal-name>
  </weblogic-enterprise-bean>
  <weblogic-enterprise-bean>
    <ejb-name>ZcmMigrationService_AsyncResponseProcessorMDB</ejb-name>
    <message-driven-descriptor>
      <pool>
        <initial-beans-in-free-pool>1</initial-beans-in-free-pool>
        <max-beans-in-free-pool>5</max-beans-in-free-pool>
      </pool>
      <destination-jndi-name>oracle.j2ee.ws.server.async.AQResponseQueue</destination-jndi-name>
      <connection-factory-jndi-name>aqjms/AsyncWSAQConnectionFactory</connection-factory-jndi-name>
    </message-driven-descriptor>
    <jndi-name>ZcmMigrationService_AsyncResponseProcessorMDB</jndi-name>
    <run-as-principal-name>FUSION_APPS_CRM_ASYNC_WS_APPID</run-as-principal-name>
  </weblogic-enterprise-bean>
</weblogic-ejb-jar>
  • Change the value of initial-beans-in-free-pool and max-beans-in-free-pool. In the case where you want to increase the number of concurrency to greater than 16, you also need to add the information of custom work manager into the deployment descriptor. We will address this later.
  • Repackage the jar:  jar -cvf AppAsyncMdb.jar META-INF/*
  • Copy back the new AppAsyncMdb.jar under /u01/app/fa/fusionapps/applications/crm/deploy/EarCustomer.ear/
  • Redeploy the EAR file

The above steps listed how to access MDB’s deployment descriptor for tuning. Don’t forget that I mentioned in the previous post: thatFusion Applications uses WebLogic’s self tuning thread pool with the default work manager, the number of threads allocated for a MDB will vary and follow this pattern: Min(max-beans-in-free-pool,16). In this rule, 16 is the maximum number of concurrency with the default work manager. See here for details.

If we want to increase the number of concurrency up to 16, for example, for the response MDB, all you need to do is to increase the number of max-beans-in-free-pool to any value up to 16 in the deployment descriptor.

However, if you want to increase the value beyond 16, you have to create a custom work manager with the appropriate thread constraints and add the reference of custom work manager into the MDB’s deployment descriptor – <dispatch-policy>Custom-WorkManager</dispatch-policy>.

The rule to get higher concurrency is: “varies between min-thread-constraint and Min(max-threads-constraint, max-beans-in-free-pool)” as described in “Determining Concurrency for WebLogic Server MDBs”.

Let’s take an example, if you want to define your maximum concurrency for the above request MDB to 25 instead of 16, the following values could be considered as a reference:

  • Create a min thread constraint = 3
  • Create a max thread constraint = 25 (assume that this thread constraint is not shared with other work managers)
  • Create a custom work manager (say CRM-request-MDB-WorkManager) in which the min-thread-constraint and the max-thread-constraint are attached
  • Set the initial-beans-in-free-pool=1
  • Set the max-beans-in-free-pool=40

To facilitate the administration, I prefer to use a global custom work manager which can be tuned dynamically at runtime. By setting the max-beans-in-free-pool to a higher value (40) greater than the value of max-thread-constraint, the maximum number of concurrency will take the value of the max-thread-constraint (25). This might avoid the modification on the deployment descriptor again if any further change to the concurrency is required (for example, increase the max-thread-constraint to 30 to get even higher concurrency).

The request MDB’s deployment descriptor of this tuning should look like the following:

  <weblogic-enterprise-bean>
    <ejb-name>ZcmMigrationService_AsyncRequestProcessorMDB</ejb-name>
    <dispatch-policy>CRM-request-MDB-WorkManager</dispatch-policy>
    <message-driven-descriptor>
      <pool>
        <initial-beans-in-free-pool>1</initial-beans-in-free-pool>
        <max-beans-in-free-pool>40</max-beans-in-free-pool>
      </pool>
      <destination-jndi-name>oracle.j2ee.ws.server.async.AQRequestQueue</destination-jndi-name>
      <connection-factory-jndi-name>aqjms/AsyncWSAQConnectionFactory</connection-factory-jndi-name>
    </message-driven-descriptor>
    <jndi-name>ZcmMigrationService_AsyncRequestProcessorMDB</jndi-name>
    <run-as-principal-name>FUSION_APPS_CRM_ASYNC_WS_APPID</run-as-principal-name>
  </weblogic-enterprise-bean>

Please document the above changes. It is likely that you will need to redo it again after any future Fusion Applications upgrade since there is not a guarantee that the changes on the MDB deployment descriptor would survive an upgrade.

Tuning JDBC

Fusion Applications uses a dedicated JDBC datasource to access the AQ queues in the database: JRFWSAsyncDSAQ.

On a given WebLogic managed server, this datasource is shared by all the MDBs – all the request MDBs and all the response MDBs. In the case where you changed the MDB concurrency as described in the above section, you might need to consider tuning the connection pool’s capacity accordingly to match the total sum of MDB concurrency.

With the JDBC monitoring described in the previous post, you can follow the WebLogic Server documentation which provides the information about how to tune JDBC connection pool to maximize the performance here.

However, please bear in mind that the same JDBC configuration is enabled on all managed servers hosting the asynchronous web services within a domain. Since the different managed servers might have different EARs with different number of MDB concurrency, the changes to the JDBC configuration will take effect on all managed servers. The JDBC settings on a specific managed server might not necessarily work properly for other managed servers. Therefore, you need to be very prudent when changing the values of JDBC configuration. A general guideline is: you can increase the max capacity, but do not ever reduce the value of Max capacity less than its original value.

In the next post, we will focus on how to troubleshooting asynchronous Web Services in Fusion Applications.

 

Add Your Comment