Unable to See Coherence MBean in Enterprise Manager Configured for WebCenter Portal Content Presenter

Introduction After configuring Coherence Cache for WebCenter Portal Content Presenter customer was not able to see the Coherence MBeans in the Enterprise Manager. Futher investigation revealed that the issue was due to configuration in WebLogic Server. Main Article After configuring Coherence Cache for WebCenter Portal Content Presenter customer was not able to see the Coherence […]

How to emulate 10g Adapter start/stop behaviour by manipulating the InboundThreadCount parameter

Introduction In 10g, there was a mechanism for suppressing consumption of messages at the Adapter level. That mechanism can not be used in 11g. But there is a way… Main Article The way to do this is to set the InboundThreadCount in the appropriate MBean to zero. This will effectively suppress consumption of messages – […]

SOA Tips & Tricks Series: Preventing OOM Exceptions with Very Large Audit Trails

In very complex flows, audit trail or flow trace sizes can grow above the limit where EM console will display this correctly with the default settings. The reason is to prevent from generating huge audit trail objects in memory and thus potentially causing OOM errors – therefore the standard limit is set to 1 MB.

You will see the following exception in a case where the audit trail exceeds this limit:

Exception occured while retrieving the Flowtrace XML for the Composite Instance; ECID: 9c37958941ffc184:-3503c179:138e19ea68a:-8000-0000000000002ee6
java.rmi.RemoteException: EJB Exception: ; nested exception is: 
   java.lang.RuntimeException: oracle.soa.management.facade.DataSetTooLargeException: Requested audit trail size is larger than threshold 1048576 chars
   at weblogic.rjvm.ResponseImpl.unmarshalReturn(ResponseImpl.java:237)
   …

 

If this is not a design or implementation error (for example an indefinite loop), then you can try to increase this audit trail limit:

instanceTrackingAuditTrailThreshold

You can modify it

in the System MBean Browser at the following location:

image

 

See also http://docs.oracle.com/cd/E25178_01/core.1111/e10108/others.htm Chapter “11.2.3 instanceTrackingAuditTrailThreshold” for details.

A Universal JMX Client for Weblogic – Monitoring BPEL Thread Pools in SOA 11g

Monitoring and optimizing BPEL Thread Pool utilization (and other metrics) is one of the key activities in performance tuning of BPEL/SOA based integrations. Although EM console provides some basic monitoring of the BPEL engine statistics, it is limited regarding the update interval, detail and the recording interval and cannot display historic data. Of course you can setup Grid Control 11g with its Repository, but this is in most cases too complex to setup just for monitoring during performance and load testing.

So, the idea came to create a tool which can easily record these statistics and export them to MS Excel or OpenOffice to create charts for the thread pool utilization over a time period (for example a whole load test execution).

All values of WLS or the SOA engine can be queried using the JMX MBean framework. I have designed the JMXClient to be able configure which MBeans should be queried by using a property file (beans.properties). I decided to connect in this first release to only one Managed Server of WLS to record / export data. This means that if you have a WLS cluster, you need to start multiple JMX Clients to record the values of each node. (In a later release the JMX Client could be optimized to query all nodes automatically).

JMXClient can be used by downloading from the project page at Sourceforge.  (including JDeveloper 11.1.1.6 project and sources)

After that you need to configure

  1. your connection properties, JAVA_HOME and WLS_HOME of  your WLS managed server of SOA in  jmxclient.bat (or jmxclient.sh)
  2. the MBeans names, WLS Server name  and the attributes to record in classes/beans.properties  (you can find the MBean names form the System MBean Browser in EM)

The syntax in jmxclient.bat is

java -cp classes;%INCLUDE_LIBS% jmxclient.JMXClient <server> <user> <password> -monitor 1000

For example

java -cp classes;%INCLUDE_LIBS% jmxclient.JMXClient 192.168.56.101 7001 weblogic welcome1 -monitor 1000

“1000” specifies the interval in milliseconds between the recording.

Then you can run it with

jmxclient > out.txt

Then simple import this text file using Excel or OpenOffice and a comma “,” as delimiter and create a line chart using line 2 as titles and lines 3 to end as data.

Let me first show a couple of results using JMXClient using the properties to record the BPEL thread pool statistics:

The following chart shows a scenario where the invoke thread pool is much too low (20) so that the queue of scheduled invocations waiting for a free thread is growing rapidly:

image

The second example shows a scenario where invoke and callback threads are within normal limits:

image

In the next posts I will show how to use JMXClient to record the BPEL process execution times or the number of messages in the AIA 11g JMS queues by simply exchanging the beans.properties file…..!

Have fun,
Stefan

DISCLAIMER: JMXClient is provided for free use “as is” without any support or warranty. Please provide enhancements or modifications you make yourself. Feedback is welcome using the comments feature of this blog.

PS: for the experts: the format of the beans.properties file:

Every line contains 3 items separated by semicolon:

  1. the name of the MBean to query
  2. the attribute to query
  3. the title string which should be displayed for the column

Example for the bpel thread pools:

oracle.dms:Location=AdminServer,name=/soainfra/engines/bpel/requests/invoke,type=soainfra_bpel_requests;active_maxValue;Invoke Active Max
oracle.dms:Location=AdminServer,name=/soainfra/engines/bpel/requests/invoke,type=soainfra_bpel_requests;scheduled_maxValue;Invoke Scheduled Max
oracle.dms:Location=AdminServer,name=/soainfra/engines/bpel/requests/invoke,type=soainfra_bpel_requests;scheduled_value;Invoke Scheduled Current
oracle.dms:Location=AdminServer,name=/soainfra/engines/bpel/requests/invoke,type=soainfra_bpel_requests;active_value;Invoke Active Value
oracle.dms:Location=AdminServer,name=/soainfra/engines/bpel/requests/invoke,type=soainfra_bpel_requests;threadCount_value;Invoke Threads Value
oracle.dms:Location=AdminServer,name=/soainfra/engines/bpel/requests/invoke,type=soainfra_bpel_requests;threadCount_maxValue;Invoke Threads Max
oracle.dms:Location=AdminServer,name=/soainfra/engines/bpel/requests/system,type=soainfra_bpel_requests;active_maxValue;System Active Max
oracle.dms:Location=AdminServer,name=/soainfra/engines/bpel/requests/system,type=soainfra_bpel_requests;scheduled_maxValue;System Scheduled Max
oracle.dms:Location=AdminServer,name=/soainfra/engines/bpel/requests/system,type=soainfra_bpel_requests;scheduled_value;System Scheduled Current
oracle.dms:Location=AdminServer,name=/soainfra/engines/bpel/requests/system,type=soainfra_bpel_requests;active_value;System Active value
oracle.dms:Location=AdminServer,name=/soainfra/engines/bpel/requests/system,type=soainfra_bpel_requests;threadCount_value;Invoke Threads Value
oracle.dms:Location=AdminServer,name=/soainfra/engines/bpel/requests/system,type=soainfra_bpel_requests;threadCount_maxValue;Invoke Threads Max
oracle.dms:Location=AdminServer,name=/soainfra/engines/bpel/requests/engine,type=soainfra_bpel_requests;active_maxValue;Engine Active Max
oracle.dms:Location=AdminServer,name=/soainfra/engines/bpel/requests/engine,type=soainfra_bpel_requests;scheduled_maxValue;Engine Scheduled Max
oracle.dms:Location=AdminServer,name=/soainfra/engines/bpel/requests/engine,type=soainfra_bpel_requests;scheduled_value;Engine Scheduled Current
oracle.dms:Location=AdminServer,name=/soainfra/engines/bpel/requests/engine,type=soainfra_bpel_requests;active_value;Engine Active value
oracle.dms:Location=AdminServer,name=/soainfra/engines/bpel/requests/engine,type=soainfra_bpel_requests;threadCount_value;Engine Threads Value
oracle.dms:Location=AdminServer,name=/soainfra/engines/bpel/requests/engine,type=soainfra_bpel_requests;threadCount_maxValue;Engine Threads Max
oracle.dms:Location=AdminServer,name=/soainfra/engines/bpel/requests/audit,type=soainfra_bpel_requests;active_maxValue;Audit Active Max
oracle.dms:Location=AdminServer,name=/soainfra/engines/bpel/requests/audit,type=soainfra_bpel_requests;scheduled_maxValue;Audit Scheduled Max
oracle.dms:Location=AdminServer,name=/soainfra/engines/bpel/requests/audit,type=soainfra_bpel_requests;scheduled_value;Audit Scheduled Max
oracle.dms:Location=AdminServer,name=/soainfra/engines/bpel/requests/audit,type=soainfra_bpel_requests;active_value;Audit Active value
oracle.dms:Location=AdminServer,name=/soainfra/engines/bpel/requests/audit,type=soainfra_bpel_requests;threadCount_value;Audit Threads Value
oracle.dms:Location=AdminServer,name=/soainfra/engines/bpel/requests/audit,type=soainfra_bpel_requests;threadCount_maxValue;Audit Threads Max

Using BPEL Performance Statistics to Diagnose Performance Bottlenecks

Tuning performance of Oracle SOA 11G applications could be challenging. Because SOA is a platform for you to build composite applications that connect many applications and “services”, when the overall performance is slow, the bottlenecks could be anyw…