SaaS – Customer Experience – Sales

Test blog entry. All content listed on this page is the property of Oracle Corp. Redistribution not allowed without written permission

Resetting Environments – An Alternate Approach

Overview There are times when you are in the midst of testing where you need to reset things back to the way they were; to be able to quickly restart your testing at from a previous point. This is technologically quite easy with current storage infrastructures supporting snapshot mechanisms. This is equally easy in those […]

Performance Management Post Go Live

Introduction A common practice for handling performance considerations during software implementations is to include a phase near the end of the development cycle for load and stress testing and to then tune the system based on the results of those tests. However, software systems are not static. The system is constantly changing through addition of […]

The Resilient Enterprise – Disaster Recovery Planning

Introduction Organizations strive to operate their business in a continuous manner with the goal of protecting their revenue, brand, and data assets. Every organization, throughout its existence, is prone to failures originating from within or without. The propensity for failures exists. A resilient organization however does two things really well. First, it implements measures to […]

A crude JMS performance comparison between different WebLogic server persistence stores

When it comes to persisting service and subsystems data (i.e. JMS), WebLogic server offers customers a choice: filesystem or relational database using JDBC.  Persistence store has implications on WLS performance and systems management.  In this post, I will provide an unofficial JMS performance comparison using different persistence stores. PLEASE NOTE: This is not an official Oracle performance benchmark.  […]

WebLogic Server Cluster Messaging Protocols

by Robert Patrick and Sabha Parameswaran WebLogic Server clusters form a loosely-federated group of managed servers that provide a model for applications to leverage for achieving scalability, load balancing, and failover. Each managed server manages its view of what servers are in the cluster and its own cluster-wide view of the JNDI tree. The cluster […]

About this site

About The A-Team and A-Team Chronicles The A-Team is a central, outbound, highly technical team comprised of Enterprise Architects, Solution Specialists and Software Engineers within the Fusion Middleware Product Development Organization that works with customers and partners, world wide, providing guidance on implementation best practices, architecture, troubleshooting and how best to use Oracle products to solve […]

OSB, Service Callouts and OQL – Part 3

In the previous sections of the “OSB, Service Callouts and OQL” series, we analyzed the threading model used by OSB for Service Callouts and analysis of OSB Server threads hung in Service callouts and identifying  the Proxies and Remote services i…

OSB, Service Callouts and OQL – Part 2

This section of the “OSB, Service Callouts and OQL” blog posting will delve into thread dump analysis of OSB server and detecting threading issues relating to Service Callout and using Heap Dump and OQL to identify the related Proxies and Business serv…

OSB, Service Callouts and OQL – Part 1

Oracle Fusion Middleware customers use Oracle Service Bus (OSB) for virtualizing Service endpoints and implementing stateless service orchestrations. Behind the performance and speed of OSB, there are a couple of key design implementations that can aff…

The curious case of SOA Human tasks’ automatic completion

A large south-Asian insurance industry customer
using Oracle BPM and SOA ran into this. I have survived this ordeal
previously myself but didnt think to blog it then. However, it seems
like a good idea to share this knowledge with this reader com…

Configuring the system for a successful Fusion Application installation – Part 1 – System limits

IntroductionI wanted to share my experience in the installation of Fusion Applications. For those that are not as familiar with it, Fusion application installation goes through several phases after the provisioning plan has been created. These arePr…

ThreadLogic version 0.95 available

  ThreadLogic version 0.95 is available for public download now. Additional Features Biggest change is support for externalizing the Advisories and group definitions. Users can use the pre-defined AdvisoryMap.xml to come up with custom …

Introducing ThreadLogic

Eric Gross and Sabha Parameswaran, from Oracle FMW Architects Team (The A-team) are happy to introduce ThreadLogic, an open source Thread Dump Analyzer tool, to Middleware community.Motivation behind ThreadLogic  The current set of TDA tools (Sam…

Analyzing Thread Dumps in Middleware – Part 4

This posting is the fourth and final section in the series Analyzing Thread Dumps in Middleware


In this section, we will see a new version of TDA (Thread Dump Analysis) tool developed by the author of this blog (Sabha Parameswaran in collaboration with his colleague, Eric Gross, also from Oracle A-Team) and its capabilities as well as some real world samples of thread dump analysis before concluding the series.


TDA A-Team – an Enhanced TDA (version 3.0)

As mentioned earlier, the current TDA tools (Samurai/TDA) don’t mine the data inside thread dumps or provide a more detailed view of what each thread is doing while just limiting themselves to reporting the state (locked/waiting/running) or the lock information. No mention of type of activity within a thread, should it be treated as normal or deserving a closer look? Can a pattern or anti-pattern be applied against them? Any possible optimizations? Are there any hot spots? Any classification of threads based on their execution cycles?

In order to fill some of the gaps in the existing TDA Tools as well as provide a detailed analysis of WebLogic Server specific thread dumps, the author of this blog (Sabha Parameswaran) decided to developed a custom TDA tool that can suggest set of actions and recommendations based on some predefined thread execution patterns.

In collaboration with Oracle A-team colleague, Eric Gross, the author decided to enhance the existing open source TDA version 2.2 instead of reinventing the wheel and leveraging the capabilities of base TDA to handle base parsing of the thread dump, the UI and Reporting. Eric Gross enhanced the TDA to fully support Sun Hotspot, JRockit (support was partial for JRockit in base TDA v2.2) and IBM JVM Thread dumps as well as adding support for custom categories in the TDA tool. Sabha Parameswaran enhanced the tool’s analytics capabilities – grouping of threads based on functionality and tagging of threads with advisories using pre-defined rules and patterns which can be extended to handle additional patterns.

We wish to thank Ingo Rockel, Robert Whitehurst and numerous others who had contributed to the original TDA which allowed us build on their work in delivering a more powerful tool for the entire Java community.

Once a thread dump is parsed and threads details are populated, each of the thread is then analyzed against matching advisories and tagged appropriately. The threads are also associated with specific Thread Groups based on functionality or thread group name.

Each of the advisory has a health level indicating severity of the issue found, pattern, name, keyword and related advice. 
Samples of advisories:
 
Thread Advisory Name Large # of WLS Muxer Threads
Health Level
WATCH
Keyword WebLogicMuxerThreads
Description Large number of WLS Muxer Threads
Advice Reduce number of WLS Muxer Threads to under 4, use -Dweblogic.SocketReaders=NoOfThreads flag in command line

 

Thread Advisory Name Stuck Thread
Health Level
FATAL
Keyword STUCK
Description Thread is Stuck, request taking very long time to finish
Advice Check why the thread or call is taking very long?? Is it blocked for unavailable or bad resource or contending for Lock? Can be ignored if it is doing repeat work in a loop (like adapter threads polling for events in a infinite loop)…
 
Thread Advisory Name Hot Spots
Health Level
WARNING
Keyword HotCallPattern
Description Multiple Threads executing same code path
Advice Ensure there are no blocking locks or bottlenecks, sufficient resources are available,remote service being invoked is responsive and scaling well to handle increased load

Each of the advisory gets triggered based on either call execution patterns observed in the thread stack or presence of other conditions (thread blocked or multiple threads blocked for same lock can trigger BlockedThreads Advisory). Sometimes a thread might be tagged as IGNORE or NORMAL based on its execution logic or might be tagged more specifically as involved in JMS send or receive client or a Servlet thread.

The health levels (in descending of severity) are FATAL (meant for Deadlocks, STUCK, Finalizer blocked etc), WARNING, WATCH (worth watching), NORMAL and IGNORE. Based on the highest criticality of threads within a group, that health level gets promoted to the Thread Group’s health level and same is repeated at the thread dump level. There can be multiple advisories tagged to a Thread, Thread Group and Thread Dump.

Snapshot of Advisory Map

Advisory Map

Snapshot of Threads tagged with advisories in the thread dump

Threads in a thread dump tagged with Advisories/Health Levels

Thread Groups Summary

The threads are associated with thread groups based on the functionality or thread names. Additional patterns exists to tag other threads (like iWay Adapter, SAP, Tibco threads) and group them. The summary page reports on health level of the group, total number of threads, threads that are blocked, critical advisories found etc.

Thread Groups Summary

Critical Advisories per thread group
The critical advisories (at Warning/Fatal health levels) found in individual threads are then promoted to the the parent thread group and reported in the thread group summary page.

Critical Advisories for Thread Group

Thread Groups

One can see the thread groups are divided into two buckets – WLS and non-WLS related threads. The JVM threads, Coherence, LDAP and other unknown custom threads go under the non-WLS bucket while all the WLS, Muxer, Oracle, SOA, JMS, Oracle Adapter threads are all under the WLS bucket.

 

 Individual Thread tagging with Advisories

Clicking on the individual threads will display the advisories and thread stack.

Advisories and details at thread level

The details of the advisory will pop up on mouse over on the advisory links.
The Advisories are color coded and  details can be highlighted.

Colored advisories for individual threads

 
Sub-groups are also created within individual Thread Groups based on Warning Levels, Hot call patterns (multiple threads executing same code section), threads doing remote I/O (socket or db reads) etc.

Following snapshot shows example of a Hot call pattern where multiple threads appear to be executing the same code path (all are attempting to update the MBean Server with a new mbean).

Hot Call Pattern – multiple threads exhibiting similar code execution

Merging of threads across multiple thread dumps and reporting of progress in the thread state

Merge has been enhanced to report on the progress of the thread across the thread dumps. Based on the order of the thread dumps, the thread stack trace is compared for every consecutive thread dump.

Merged view showing progress information for individual threads  

Merged reporting of individual thread stack traces (exists in base TDA version 2.2)

Merged Thread stack traces across thread dumps

Merging can also be done across multiple thread dump log files (like in case of IBM which creates new log file containing the thread dump every time a request is issued).

Usability benefits of new TDA A-Team
 
Thanks to the advisories and health levels, its easy for users to quickly understand the usage patterns, hot spots, thread groups, as well as highlight the patterns or anti-patterns already implemented in the advisory list.

For example of an anti-pattern: a Servlet thread should not be waiting for an event to occur as this will translate to bad performance for end user. Similarly usage of synchronous jms consumers might be less performant compared to using async consumers. Too many WLS Muxer threads is not advisable. If WLS Muxer or Finalizer threads are blocked for unrelated locks, this will be a fatal condition. It would be okay to ignore STUCK warning issued by WLS Server for Pollers like AQ Adapter threads but not for other threads that are handling servlet request.

The thread groups help in bunching together related threads; so SOA Suite users can see how many BPEL Invoke and Engine threads are getting used, B2B users can see number of JMS consumers/producers, WLS users can look at condition and health of Muxer threads, similarly for JVM/Coherence/LDAP/other thread groups.

The merged report lets the user see at a glance the critical threads and check if they are progressing or not instead of  wading through numerous threads and associated thread dumps.

We hope this tool can really help both beginners and experts do their jobs more quickly and efficiently when it comes to thread dumps. 

Real World Sample of Thread Dump Analysis

A Customer was planning on a POC (Proof of Concept) of WebLogic Portal Server (WLP) on the new Oracle Exalogic Platform and measure its performance relative to their existing configurations. The test configuration used a cluster of WLP servers, communicating via Multicast for cluster membership and other service announcements. As soon as the servers were brought up, the servers became unresponsive. The server instances could not serve any request – neither traffic to Portal Application nor WLS Admin Server requests for status from the server instances. Following errors get getting thrown in the server: 

UnsyncCircularQueue$FullQueueException: Queue exceeded maximum capacity of: ‘65536’ elements. 


There were also frequent log messages of p13n cache getting updated and flushed repeatedly. The p13n cache is the Personalization cache used internally by Portal for handling Portal personalization. Each of the servers in the cluster are kept of changes in cache via multicast broadcasts. When the customer disabled the p13n cache, the server regained normalcy and responsiveness. But the p13n cache is a requirement for portal application functionality and its not possible to have it disabled.

The FullQueueException essentially indicated that the WLS’s internal request queue was getting overwhelmed with requests and its starting to ignore/drop new requests as its already has 64K requests waiting to be executed. These requests can be internal (like transaction timeouts, schedulers, async processing) or externally generated (like client http requests). The storm of requests coupled with p13n being a key presence for the issue indicated p13n cache is triggering an internal flood of requests to the server somehow. It was decided to take thread dumps to understand what the p13n cache is executing.

Analyzing the thread dumps threw light on a weird behavior.

    at java/lang/Thread.sleep(J)V(Native Method)
at weblogic/cluster/MulticastFragmentSocket.sendThrottled(MulticastFragmentSocket.java:198)
at weblogic/cluster/MulticastFragmentSocket.send(MulticastFragmentSocket.java:157)
^-- Holding lock: weblogic/cluster/MulticastFragmentSocket@0x1d4afd4c8[thin lock]
at weblogic/cluster/FragmentSocketWrapper.send(FragmentSocketWrapper.java:91)
at weblogic/cluster/MulticastSender.fragmentAndSend(MulticastSender.java:395)
at weblogic/cluster/MulticastSender.send(MulticastSender.java:178)
^-- Holding lock: weblogic/cluster/MulticastSender@0x1c7ce3628[thin lock]
at com/bea/p13n/cache/internal/system/SystemCacheManager.doFlushCluster(SystemCacheManager.java:222)
at com/bea/p13n/cache/internal/system/SystemCacheManager.flushClusterKeys(SystemCacheManager.java:193)
at com/bea/p13n/cache/CacheManager.flushKeys(CacheManager.java:117)

at com/bea/p13n/cache/internal/system/SystemCacheManager.doFlushLocal(SystemCacheManager.java:135)
at com/bea/p13n/cache/internal/system/SystemCacheManager.flushLocalKeys(SystemCacheManager.java:84)
at com/bea/p13n/cache/internal/system/CacheClusterMessage.execute(CacheClusterMessage.java:80)
at weblogic/cluster/MulticastReceiver$1.run(MulticastReceiver.java:112)
at weblogic/work/ExecuteThread.execute(ExecuteThread.java:201)
at weblogic/work/ExecuteThread.run(ExecuteThread.java:173)

The above thread stack indicates the MulticastReceiver used by WLS internally to handle multicast announcements got some request/announcement that gets handled by the p13n cache. It appears to trigger a flush of local keys in the cache. But then it gets modified as a flush of cluster keys which then results in a MulticastSender sending out a cluster broadcast.

This essentially means for every announcement that lands on a server instance, it will resend the announcement to other members whcih then gets replicated for every member across the cluster leading to an infinite looping of the same message and contributing to a never ending multicast packet storm.

Analyzing the product code against the stack trace, it appeared that there was a mismatch between the CacheManager.flushKeys() implementation as the method code lines didn’t match. So, it was apparent that there was some variant form of CacheManager that was getting picked by the application while handling the cluster announcements.

    at java/lang/Thread.sleep(J)V(Native Method)
at weblogic/cluster/MulticastFragmentSocket.sendThrottled(MulticastFragmentSocket.java:198)
at weblogic/cluster/MulticastFragmentSocket.send(MulticastFragmentSocket.java:157)

REPEAT MULTICAST TRAFFIC - CREATE INFINITE FLOOD

^-- Holding lock: weblogic/cluster/MulticastFragmentSocket@0x1d4afd4c8[thin lock]
at weblogic/cluster/FragmentSocketWrapper.send(FragmentSocketWrapper.java:91)
at weblogic/cluster/MulticastSender.fragmentAndSend(MulticastSender.java:395)
at weblogic/cluster/MulticastSender.send(MulticastSender.java:178)

^-- Holding lock: weblogic/cluster/MulticastSender@0x1c7ce3628[thin lock]

LOGIC HIJACKED
at com/bea/p13n/cache/internal/system/SystemCacheManager.doFlushCluster(SystemCacheManager.java:222)
at com/bea/p13n/cache/internal/system/SystemCacheManager.flushClusterKeys(SystemCacheManager.java:193)
at com/bea/p13n/cache/CacheManager.flushKeys(CacheManager.java:117)

LOCAL FLUSH ONLY
at com/bea/p13n/cache/internal/system/SystemCacheManager.doFlushLocal(SystemCacheManager.java:135)
at com/bea/p13n/cache/internal/system/SystemCacheManager.flushLocalKeys(SystemCacheManager.java:84)
at com/bea/p13n/cache/internal/system/CacheClusterMessage.execute(CacheClusterMessage.java:80)

RECEIVE MULTICAST MESSAGE
at weblogic/cluster/MulticastReceiver$1.run(MulticastReceiver.java:112)
at weblogic/work/ExecuteThread.execute(ExecuteThread.java:201)
at weblogic/work/ExecuteThread.run(ExecuteThread.java:173)

Using WLS ClassLoader Analysis Tool (CAT), we were able to drill down to where this particular CacheManager class was getting loaded from. It was packaged inside a jar that got added to the Portal EAR application libraries. Removing that particular jar containing the CacheManager resolved the problem. Later investigation revealed that the jar came from an older version of the Portal Product and should not have been used with the latest version of Portal that was being tested on Exalogic. The analysis of a single thread stack trace helped pin point the root cause for all the hang conditions in this scenario.

Limitations of Thread Dumps

Although Thread dumps can throw light on multiple things, there are also some limitations.

  • It is a snapshot of a running system
    • The system should have some representative load or exhibit symptoms when capturing thread dumps
    • The dumps are of no value if none of the above
  • It cannot show details on memory usage
  • It can show what is the pattern executed but not how many times it was executed previously (unless there are multiple threads dumps captured at periodic intervals).
  • It cannot show details of the request that is driving the code execution or which remote service is slow
    • Same set of framework code can get executed for SOA Suite BPEL Processes or OSB Proxies
    • we wont be able to determine which external service is blocking the response for a thread
    • Would require analysis of server logs/proxies/access logs and or heap dumps to identify slow services and then drill down from there or other forms of monitoring (EM/OSB Console monitoring statistics)
Thread Dump takeaways
 
Even if thread dump does not give a direct solution, it can definitely provide good pointers as to where the problem lies and what are the hotspots.
 
  • Finalizer doing heavy work (too many finalizable objects to clean up) in each successive thread dump would indicate memory handling issues (and possibly overuse/misuse of finalize – lazy cleanup or recreating additional objects during finalize instead of active deleting/cleaning up)
  • Busy GC Threads would indicate memory thrashing
  • Threads stuck on socket reads would indicate slow reads or backends not responding in a timely fashion or socket closures due to firewall or other bad socket handling
  • Threads stuck in socket connect indicates the backend service itself is unavailable or not listening at that address or wrong connection detail or connection recreation every time.
  • Threads stuck in database sql reads/executes can indicate db response issue or sql tuning issue
  • Threads blocked in opening files, jars, zips or checking on file attributes can indicate problem in reading the application bits or files on a remote storage or network mount point that’s will slow down server startup/deployment. One customer problem was related to server instance accessing WLS install & deployed application bits from a remote site that was about a hundred miles away.
  • Threads blocked on locks for synchronized code imply bottlenecks
    • Resolve the bottleneck by increasing the resources under contention
    • Avoid the synchronized calls entirely (ex: logging might be excessively synchronized, reduce logging levels or turn off logging ) whenever possible
    • Optimize the call execution to reduce lock time
  • Some of the issues might then require further investigation into environment (process, cpu, memory, io, network etc) or services/partners/actors/clients to resolve the issues
  • For cluster related or distributed system issues, thread dumps should be captured and analyzed from all related parties (cluster/services). 
 
 
Summary
 
Thread Dump Analysis is a really valuable and key weapon in every Java Developer and Architect’s arsenal when it comes to understanding, identifying and resolving performance or other issues. I hope this series (along with the new enhanced TDA tool) helps makes it easier and useful for people to troubleshoot, tune performance and get a better handle over complex Java/JEE systems.

Analyzing Thread Dumps in Middleware – Part 3

This posting is the third section in the series Analyzing Thread Dumps in MiddlewareThis section details with tools for analyzing thread dumps along with limitations of such tools.Thread Dump Analysis Tools Earlier in the series, we saw how to ca…

Analyzing Thread Dumps in Middleware – Part 2

This posting is the second section in the series Analyzing Thread Dumps in Middleware

This section details with when, how to capture and analyze thread dumps with special focus on WebLogic Application Server related thread dumps. Subsequent sections will deal with more real-world samples and tools for automatic analysis of Thread Dumps.

The Diagnosis

Everyone must have gone through periodic health checkups. As a starting point, Doctors always order for a blood test. Why this emphasis on blood test? Can’t they just go by the patient’s feedback or concerns? What is special about blood test?

Blood Test help via:

  • Capturing a snapshot of the overall health of the body (cells/antibodies/…)
  • Detecting Abnormalities (low/high hormones, elevated blood sugar levels)
  • Identifying and start focusing  on the problem areas with prevention and treatment.

The Thread Dump is the equivalent of a blood test for a JVM. It is a state dump of the running JVM and all its parts that are executing at that moment in time.

  • It can help identify current execution patterns and help focus attention on relevant parts
    • There might be 100s of components and layers but difficult to identify what is getting invoked the most and where to focus attention. A thread dump taken during load will highlight the most executed code paths.
  • Dependencies and bottlenecks in code and application design
  • Show pointers for enhancements and optimizations

When to capture Thread Dumps

Under what conditions should we capture thread dumps? Anytime or specific times? Capturing thread dumps are ideal for following conditions:

  1. To understand what is actively executed in the system when under load
  2. When system is sluggish or slow or hangs
  • Java Virtual Process running but Server or App itself not responding
  • Pages take forever to load
  • Response time increases
  • Application and or Server behavior erratic
  • Just to observe hot spots (frequently executed code) under load
  • For Performance tuning
  • For Dependency analysis
  • For bottleneck identification and resolution
  • To capture snapshot of running state in server

 

The cost associated with capturing thread dumps is close to near zero; Java Profilers and other tools can consume anywhere from 5 to 20% overhead while a thread dump is more of  a snapshot of threads which requires no real heavy weight processing on part of the JVM. There can be some cost only if there are too many interrupts to the JVM within real short intervals (like dumps forever every second or so).

How to capture Thread Dumps

There are various options to capture thread dumps. JVM vendor tools allow capture and gather of thread dumps. Operating System interrupt signals to the process can also be used in generating thread dumps.

Sun Hotspot’s jstack tool (under JAVA_HOME or JRE Home/bin) can generate thread dumps given the jvm process id. Similarly, jrcmd (from JRockit Mission Control) can generate thread dumps given a jvm process id and using the print_threads command. Both require to be run on the same operating system as the jvm process and both dump the output to the user shell.

In Unix, kill -3 <PID> or kill -SIGQUIT<PID> will generate thread dumps on the JVM process. But the output of the thread dump will go to the STDERR. Unless the STDERR was redirected to a log file, the thread dump cannot be saved when issued via kill signals. Ensure the process is always started and STDERR is redirected to a log (best practice to also have STDOUT redirected to same log file as STDERR).

In Windows, Ctrl-Break to the JVM process running in foreground can generate thread dumps. The primary limitation is the process needs to be running in the shell. Process Explorer in windows can also help in generating thread dumps but its much more problematic to get all the thread stacks in one shot. One has to wade through each thread and gets its stack. Another thing to keep in mind is, JVM might ignore interrupts if they were started with -Xrs flag.

WebLogic Application Server allows capture of thread dumps via additional options:

    1. WLS Admin Console can generate thread dumps. Go to the Server Instance -> Monitoring -> Threads -> Dump Threads option.
    2. Using weblogic’s WLST (WebLogic Scripting Tool) to issue Thread dumps.

 

Analyzing Thread Dumps in Middleware – Part 1

This posting deals with analysis of Thread dumps, for improving Middleware Performance (at App Server or Application level) as well as for general troubleshooting. It will also go into details of WebLogic Application Server specific Thread Dump Analysis and fine tuning.

This section details with basics of thread states and locking in general. Subsequent sections will deal with capturing thread dumps and analysis with particular emphasis on WebLogic Application Server Thread dumps.

Thread Dumps

A Thread Dump is a brief snapshot in textual format of threads within a Java Virtual Machine (JVM). This is equivalent to process dump in the native world. Data about each thread including the name of the thread, priority, thread group, state (running/blocked/waiting/parking) as well as the execution stack in form of thread stack trace is included in the thread dump. All threads – the Java VM threads (GC threads/scavengers/monitors/others) as well as application and server threads are all included in the dump. Newer versions of JVMs also report blocked thread chains (like ThreadA is locked for a resource held by ThreadB) as well as deadlocks (circular dependency among threads for locks).

Different JVM Vendors display the data in different formats (markers for start/end of thread dumps, reporting of locks and thread states, method signatures) but the underlying data exposed by the thread dumps remains the same across vendors.

Sample of a JRockit Thread Dump:


===== FULL THREAD DUMP ===============
Mon Feb 06 11:38:58 2012
Oracle JRockit(R) R28.0.0-679-130297-1.6.0_17-20100312-2123-windows-ia32

"Main Thread" id=1 idx=0x4 tid=4184 prio=5 alive, in native
    at java/net/PlainSocketImpl.socketConnect(Ljava/net/InetAddress;II)V(Native Method)
    at java/net/PlainSocketImpl.doConnect(PlainSocketImpl.java:333)
    ^-- Holding lock: java/net/SocksSocketImpl@0x10204E50[biased lock]
    at java/net/PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)
    at java/net/PlainSocketImpl.connect(PlainSocketImpl.java:182)
    at java/net/SocksSocketImpl.connect(SocksSocketImpl.java:366)
    at java/net/Socket.connect(Socket.java:525)
    at java/net/Socket.connect(Socket.java:475)
    at sun/net/NetworkClient.doConnect(NetworkClient.java:163)
    at sun/net/www/http/HttpClient.openServer(HttpClient.java:394)
    at sun/net/www/http/HttpClient.openServer(HttpClient.java:529)
    ^-- Holding lock: sun/net/www/http/HttpClient@0x10203FB8[biased lock]
    at sun/net/www/http/HttpClient.&lt;init&gt;(HttpClient.java:233)
    at sun/net/www/http/HttpClient.New(HttpClient.java:306)
    at sun/net/www/http/HttpClient.New(HttpClient.java:323)
    at sun/net/www/protocol/http/HttpURLConnection.getNewHttpClient(HttpURLConnection.java:860)
    at sun/net/www/protocol/http/HttpURLConnection.plainConnect(HttpURLConnection.java:801)
    at sun/net/www/protocol/http/HttpURLConnection.connect(HttpURLConnection.java:726)
    at sun/net/www/protocol/http/HttpURLConnection.getOutputStream(HttpURLConnection.java:904)
    ^-- Holding lock: sun/net/www/protocol/http/HttpURLConnection@0x101FAD88[biased lock]
    at post.main(post.java:29)
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace

"(Signal Handler)" id=2 idx=0x8 tid=4668 prio=5 alive, daemon

"(OC Main Thread)" id=3 idx=0xc tid=6332 prio=5 alive, native_waiting, daemon

"(GC Worker Thread 1)" id=? idx=0x10 tid=1484 prio=5 alive, daemon

"(GC Worker Thread 2)" id=? idx=0x14 tid=5548 prio=5 alive, daemon

"(Code Generation Thread 1)" id=4 idx=0x30 tid=8016 prio=5 alive, native_waiting, daemon

"(Code Optimization Thread 1)" id=5 idx=0x34 tid=3596 prio=5 alive, native_waiting, daemon

"(VM Periodic Task)" id=6 idx=0x38 tid=1352 prio=10 alive, native_blocked, daemon

"(Attach Listener)" id=7 idx=0x3c tid=6592 prio=5 alive, native_blocked, daemon

"Finalizer" id=8 idx=0x40 tid=1576 prio=8 alive, native_waiting, daemon
    at jrockit/memory/Finalizer.waitForFinalizees(J[Ljava/lang/Object;)I(Native Method)
    at jrockit/memory/Finalizer.access$700(Finalizer.java:12)
    at jrockit/memory/Finalizer$4.run(Finalizer.java:183)
    at java/lang/Thread.run(Thread.java:619)
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace

"Reference Handler" id=9 idx=0x44 tid=3012 prio=10 alive, native_waiting, daemon
    at java/lang/ref/Reference.waitForActivatedQueue(J)Ljava/lang/ref/Reference;(Native Method)
    at java/lang/ref/Reference.access$100(Reference.java:11)
    at java/lang/ref/Reference$ReferenceHandler.run(Reference.java:82)
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace

"(Sensor Event Thread)" id=10 idx=0x48 tid=980 prio=5 alive, native_blocked, daemon

"VM JFR Buffer Thread" id=11 idx=0x4c tid=6072 prio=5 alive, in native, daemon

===== END OF THREAD DUMP ===============

Sample of a Sun Hotspot Thread Dump (executing same code as above)

2012-02-06 11:37:30

Full thread dump Java HotSpot(TM) Client VM (16.0-b13 mixed mode):

"Low Memory Detector" daemon prio=6 tid=0x0264bc00 nid=0x520 runnable [0x00000000]
   java.lang.Thread.State: RUNNABLE

"CompilerThread0" daemon prio=10 tid=0x02647400 nid=0x1ae8 waiting on condition [0x00000000]
   java.lang.Thread.State: RUNNABLE

"Attach Listener" daemon prio=10 tid=0x02645800 nid=0x1480 runnable [0x00000000]
   java.lang.Thread.State: RUNNABLE

"Signal Dispatcher" daemon prio=10 tid=0x02642800 nid=0x644 waiting on condition [0x00000000]
   java.lang.Thread.State: RUNNABLE

"Finalizer" daemon prio=8 tid=0x02614800 nid=0x1e70 in Object.wait() [0x1882f000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on &lt;0x04660b18&gt; (a java.lang.ref.ReferenceQueue$Lock)
        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
        - locked &lt;0x04660b18&gt; (a java.lang.ref.ReferenceQueue$Lock)
        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
        at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)

"Reference Handler" daemon prio=10 tid=0x02610000 nid=0x1b84 in Object.wait() [0x1879f000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on &lt;0x04660a20&gt; (a java.lang.ref.Reference$Lock)
        at java.lang.Object.wait(Object.java:485)
        at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
        - locked &lt;0x04660a20&gt; (a java.lang.ref.Reference$Lock)

"main" prio=6 tid=0x00ec9400 nid=0x19e4 runnable [0x0024f000]
   java.lang.Thread.State: RUNNABLE
        at java.net.PlainSocketImpl.socketConnect(Native Method)
        at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)
        - locked <0x04642958> (a java.net.SocksSocketImpl)
        at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)
        at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
        at java.net.Socket.connect(Socket.java:525)
        at java.net.Socket.connect(Socket.java:475)
        at sun.net.NetworkClient.doConnect(NetworkClient.java:163)
        at sun.net.www.http.HttpClient.openServer(HttpClient.java:394)
        at sun.net.www.http.HttpClient.openServer(HttpClient.java:529)
        - locked <0x04642058> (a sun.net.www.http.HttpClient)
        at sun.net.www.http.HttpClient.<init>(HttpClient.java:233)
        at sun.net.www.http.HttpClient.New(HttpClient.java:306)
        at sun.net.www.http.HttpClient.New(HttpClient.java:323)
        at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:860)
        at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:801)
        at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:726)
        at sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:904)
        - locked <0x04639dd0> (a sun.net.www.protocol.http.HttpURLConnection)

"VM Thread" prio=10 tid=0x0260d000 nid=0x4dc runnable

"VM Periodic Task Thread" prio=10 tid=0x02656000 nid=0x16b8 waiting on condition

JNI global references: 667

Heap
 def new generation   total 4928K, used 281K [0x04660000, 0x04bb0000, 0x09bb0000)
  eden space 4416K,   6% used [0x04660000, 0x046a6460, 0x04ab0000)
  from space 512K,   0% used [0x04ab0000, 0x04ab0000, 0x04b30000)
  to   space 512K,   0% used [0x04b30000, 0x04b30000, 0x04bb0000)
 tenured generation   total 10944K, used 0K [0x09bb0000, 0x0a660000, 0x14660000)
   the space 10944K,   0% used [0x09bb0000, 0x09bb0000, 0x09bb0200, 0x0a660000)
 compacting perm gen  total 12288K, used 1704K [0x14660000, 0x15260000, 0x18660000)
   the space 12288K,  13% used [0x14660000, 0x1480a290, 0x1480a400, 0x15260000)
No shared spaces configured.

 

 

Sample of  an IBM Thread dump

NULL ------------------------------------------------------------------------
0SECTION THREADS subcomponent dump routine
NULL =================================
NULL
1XMCURTHDINFO Current Thread Details
NULL ----------------------
NULL
1XMTHDINFO All Thread Details
NULL ------------------
NULL
2XMFULLTHDDUMP Full thread dump J9 VM (J2RE 6.0 IBM J9 2.4 Windows Vista x86-32 build jvmwi3260sr4ifx-20090506_3499120090506_034991_lHdSMr,
native threads):
3XMTHREADINFO "main" TID:0x00554B00, j9thread_t:0x00783AE4, state:CW, prio=5
3XMTHREADINFO1 (native thread ID:0x1E48, native priority:0x5, native policy:UNKNOWN)
4XESTACKTRACE at com/ibm/oti/vm/BootstrapClassLoader.loadClass(BootstrapClassLoader.java:65)
4XESTACKTRACE at sun/net/NetworkClient.isASCIISuperset(NetworkClient.java:122)
4XESTACKTRACE at sun/net/NetworkClient.<clinit>(NetworkClient.java:83)
4XESTACKTRACE at java/lang/J9VMInternals.initializeImpl(Native Method)
4XESTACKTRACE at java/lang/J9VMInternals.initialize(J9VMInternals.java:200(Compiled Code))
4XESTACKTRACE at java/lang/J9VMInternals.initialize(J9VMInternals.java:167(Compiled Code))
4XESTACKTRACE at sun/net/www/protocol/http/HttpURLConnection.getNewHttpClient(HttpURLConnection.java:783)
4XESTACKTRACE at sun/net/www/protocol/http/HttpURLConnection.plainConnect(HttpURLConnection.java:724)
4XESTACKTRACE at sun/net/www/protocol/http/HttpURLConnection.connect(HttpURLConnection.java:649)
4XESTACKTRACE at sun/net/www/protocol/http/HttpURLConnection.getOutputStream(HttpURLConnection.java:827)
4XESTACKTRACE at post.main(post.java:29)
3XMTHREADINFO "JIT Compilation Thread" TID:0x00555000, j9thread_t:0x00783D48, state:CW, prio=10
3XMTHREADINFO1 (native thread ID:0x111C, native priority:0xB, native policy:UNKNOWN)
3XMTHREADINFO "Signal Dispatcher" TID:0x6B693300, j9thread_t:0x00784210, state:R, prio=5
3XMTHREADINFO1 (native thread ID:0x1E34, native priority:0x5, native policy:UNKNOWN)
4XESTACKTRACE at com/ibm/misc/SignalDispatcher.waitForSignal(Native Method)
4XESTACKTRACE at com/ibm/misc/SignalDispatcher.run(SignalDispatcher.java:54)
3XMTHREADINFO "Gc Slave Thread" TID:0x6B693800, j9thread_t:0x0078EABC, state:CW, prio=5
3XMTHREADINFO1 (native thread ID:0x1AA4, native priority:0x5, native policy:UNKNOWN)
3XMTHREADINFO "Gc Slave Thread" TID:0x6B695500, j9thread_t:0x0078ED20, state:CW, prio=5
3XMTHREADINFO1 (native thread ID:0x14F8, native priority:0x5, native policy:UNKNOWN)
3XMTHREADINFO "Gc Slave Thread" TID:0x6B695A00, j9thread_t:0x0078EF84, state:CW, prio=5
3XMTHREADINFO1 (native thread ID:0x9E0, native priority:0x5, native policy:UNKNOWN)
3XMTHREADINFO "Gc Slave Thread" TID:0x6B698800, j9thread_t:0x0078F1E8, state:CW, prio=5
3XMTHREADINFO1 (native thread ID:0x1FB8, native priority:0x5, native policy:UNKNOWN)
3XMTHREADINFO "Gc Slave Thread" TID:0x6B698D00, j9thread_t:0x0078F44C, state:CW, prio=5
3XMTHREADINFO1 (native thread ID:0x1A58, native priority:0x5, native policy:UNKNOWN)
3XMTHREADINFO "Gc Slave Thread" TID:0x6B69BB00, j9thread_t:0x0078F6B0, state:CW, prio=5
3XMTHREADINFO1 (native thread ID:0x1430, native priority:0x5, native policy:UNKNOWN)
3XMTHREADINFO "Gc Slave Thread" TID:0x6B69C000, j9thread_t:0x029D8FE4, state:CW, prio=5
3XMTHREADINFO1 (native thread ID:0xBC4, native priority:0x5, native policy:UNKNOWN)
NULL ------------------------------------------------------------------------
Basics of Thread States
 
A Thread can be in one of the following states:
 
  1. New
  2. Runnable
  3. Non-Runnable 
  • Sleep for a time duration
  • Wait for a condition/event
  • Blocked for a lock
  • Dead

 Following image shows the transition between thread states.

In a thread dump, we are looking at threads that have been created already and are either in running or non-running states. So, the new (unless a new thread just got created at the exact moment the thread dump was generated) & dead states are really not of value or present in a thread dump.

Running state implies the thread is actively working on something. Coming to the non-runnable states, its possible a thread has nothing to do and sleeps for some duration and periodically checks for condition to start work. Wait for a condition implies the thread is waiting for some form of notification or an event and can start work once there is a green light. Its much more efficient to use waiting for a condition pattern instead of regular sleep-wake up pattern for optimal usage of resources. If there are multiple threads regularly doing a sleep and wake up periodically, they can be optimized to wake up on notify of an event and only one thread would be successful in getting the notify call instead of all doing the regular check for event in the sleep pattern.

Blocked implies it cannot proceed with its work till it can obtain a lock which is currently held by someone else. This is similar to obtaining a lock to a Critical Region or Semaphore (in OS semantics) before proceeding with the work.

States

Each of the thread entry in a thread dump specifies the state along with name and priority.

In Sun Hotspot, the state is also part of the individual thread entry. The main thread appears as RUNNABLE while the Finalizer GC thread appears in a WAITING state.



"main" prio=6 tid=0x00ec9400 nid=0x19e4 runnable [0x0024f000]
   java.lang.Thread.State: RUNNABLE
        at java.net.PlainSocketImpl.socketConnect(Native Method)
"Finalizer" daemon prio=8 tid=0x02614800 nid=0x1e70 in Object.wait() [0x1882f000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on &lt;0x04660b18&gt; (a java.lang.ref.ReferenceQueue$Lock)



In a JRockit thread dump, there is no state mentioned for running thread. The Finalizer appears in native_waiting state.

"Main Thread" id=1 idx=0x4 tid=4184 prio=5 alive, in native
    at java/net/PlainSocketImpl.socketConnect(Ljava/net/InetAddress;II)V(Native Method)
    at java/net/PlainSocketImpl.doConnect(PlainSocketImpl.java:333)
"Finalizer" id=8 idx=0x40 tid=1576 prio=8 alive, native_waiting, daemon
    at jrockit/memory/Finalizer.waitForFinalizees(J[Ljava/lang/Object;)I(Native Method)
    at jrockit/memory/Finalizer.access$700(Finalizer.java:12)

In IBM thread dump, the state is specified by the field state. CW stands for Condition Wait.

"main" TID:0x00554B00, j9thread_t:0x00783AE4, state:CW, prio=5
3XMTHREADINFO1 (native thread ID:0x1E48, native priority:0x5, native policy:UNKNOWN)
4XESTACKTRACE at com/ibm/oti/vm/BootstrapClassLoader.loadClass(BootstrapClassLoader.java:65)
4XESTACKTRACE at sun/net/NetworkClient.isASCIISuperset(NetworkClient.java:122)
4XESTACKTRACE at sun/net/NetworkClient.<clinit>(NetworkClient.java:83)
....................
4XESTACKTRACE at sun/net/www/protocol/http/HttpURLConnection.getOutputStream(HttpURLConnection.java:827)
4XESTACKTRACE at post.main(post.java:29)
3XMTHREADINFO "JIT Compilation Thread" TID:0x00555000, j9thread_t:0x00783D48, state:CW, prio=10
3XMTHREADINFO1 (native thread ID:0x111C, native priority:0xB, native policy:UNKNOWN)
3XMTHREADINFO "Gc Slave Thread" TID:0x6B693800, j9thread_t:0x0078EABC, state:CW, prio=5
3XMTHREADINFO1 (native thread ID:0x1AA4, native priority:0x5, native policy:UNKNOWN)
 



Locks

What are locks? Locks are regions that act as speed beakers or gatekeepers to ensure only one thread can obtain a temporary ownership of a resource and start work on something. This is mainly to ensure multiple threads don’t work in the same region and mess up the final outcome or to ensure ordering of execution. This can be equated to only one person can operate on an ATM machine at a time as you don’t want multiple different users to withdraw or deposit at the same time from the same machine without the ATM machine being able to confirm each of the operation being carried out (like credit multiple times a single deposit or credit to wrong accounts). Similar to Writer/Readers problem in OS scheduling, we don’t want the multiple writers to intersperse their writes to the same page or readers to read incomplete content. JVM provides implicit locks whenever a code demarcates a method call as synchronized or a region of code within a method. The lock can be on an instance or class level or method level. JDK 1.6 provides higher level abstractions in form of concurrent.locks package (Rentrant Locks) similar to the jvm locks.

What happens when a thread requests for a Lock? If the lock is not owned by anyone, the thread becomes the new owner till it relinquishes it. What if another thread attempts to obtain ownership of the same lock when its already owned by a different thread? The new bidder gets added to a blocked list of contenders for the lock. As more threads join the waiting contenders list, the chances of getting ownership decreases among them. Normally the owner of the lock might be done finishing its job in a short duration and would relinquish the lock in a short while and one of the threads from the blocked list is chosen to become the new owner. But if the owner has to do heavy weight lifting and continues to own the lock for lot longer, and the lock is required by multiple threads, this can create a bottleneck in application or server execution as other threads cannot proceed without getting the lock which is held by the long or slow running owner. This can lead to blocked thread chains.

JRockit Thread blocking:


"ExecuteThread: '13' for queue: 'weblogic.kernel.Default (self-tuning)'" id=131 idx=0x248 tid=8047 prio=5 alive, blocked, native_blocked, daemon
-- Blocked trying to get lock: weblogic/utils/classloaders/GenericClassLoader@0xd1bacb10[fat lock]
at jrockit/vm/Threads.waitForUnblockSignal()V(Native Method)
at jrockit/vm/Locks.fatLockBlockOrSpin(Locks.java:1411)
at jrockit/vm/Locks.lockFat(Locks.java:1512)
at jrockit/vm/Locks.monitorEnterSecondStageHard(Locks.java:1054)[optimized]
at jrockit/vm/Locks.monitorEnterSecondStage(Locks.java:1005)[optimized]
at jrockit/vm/Locks.monitorEnter(Locks.java:2179)[optimized]
at java/lang/ClassLoader.loadClass(ClassLoader.java:292)
at java/lang/ClassLoader.loadClass(ClassLoader.java:248)
at weblogic/utils/classloaders/GenericClassLoader.loadClass(GenericClassLoader.java:179)

Sun Hotspot Thread blocking:


"ExecuteThread: '33' for queue: 'weblogic.kernel.Default (self-tuning)'"
waiting for lock java.util.Collections$SynchronizedSet@1dc8b68c BLOCKED
weblogic.management.provider.internal.RegistrationManagerImpl.invokeRegistrationHandlers(RegistrationManagerImpl.java:211)
weblogic.management.provider.internal.RegistrationManagerImpl.unregister(RegistrationManagerImpl.java:105)
weblogic.management.runtime.RuntimeMBeanDelegate.unregister(RuntimeMBeanDelegate.java:289)
weblogic.messaging.common.PrivilegedActionUtilities$2.run(PrivilegedActionUtilities.java:56)
weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:363)
weblogic.security.service.SecurityManager.runAs(SecurityManager.java:147)

What if ThreadA owning LockA now needs LockB which is held by ThreadB. ThreadA will block till it gets ownership of LockB. Everyone else waiting for LockA will continue to block for LockA till ThreadA releases it. Deadlock is the condition that occurs if ThreadA is blocked for LockB while holding LockA and ThreadB is also blocked for LockA while holding LockB. Its a mutual deadlock where neither thread can proceed as its blocked for the other to release a resource it needs for its completion. The level of circular dependency can be simply between two thread or more threads. Restart of the JVM is the only way to clear the Deadlock.

Synchronization and Wait


A thread might acquire a lock but find itself unable to proceed as it has to wait for a condition to happen (like plane boarded and ready to fly but has to wait for take off signal from Control tower). In those cases, there might be others waiting to obtain lock owned by it. The owner thread can do a wait on the lock object which will make it relinquish the lock voluntarily and place itself  on the waiting list till it gets a notification that has re-obtained a lock and rechecks the condition before proceeding ahead.

The usage pattern is for the thread that has changed the condition to obtain a lock and call notify on it before releasing the lock so the waiters can obtain the lock and check for the condition.

The following table shows three threads working on the same lock object but doing different activities.

Thread1 acquires lock on synchronized code segment and then waits for event notification.

Will show up as having locked, released and now waiting for notification on the same Lock object (LockObj@0x849bb)

void waitForRequest() {
// Acquire lock on lockObj
synchronized(lockObj) {
doSomething1();
while (!condition) {
// relinquish lock voluntarily
lockObj.wait();
}
doRestOfStuff();
}
//lock released
}

"ExecuteThread: ‘1‘
-- Waiting for notification on: LockObj@0x849bb
at Threads.waitForNotifySignal
at java/lang/Object.wait()
at ExThread.waitForRequest()
^-- Lock released while waiting: LockObj@0x849bb
Thread2 acquires lock on synchronized code segment and then sends notification Will show as holding (or locked) Lock object (LockObj@0x849bb)
void fillRequest() {
// Acquire lock on lockObj
synchronized(lockObj) {
doSomething2();
condition = true;
// notify on lock waiters
lockObj.notify();
}
//lock released
}

"ExecuteThread: ‘2‘
at doSomething2()
-- Holding lock on: LockObj@0x849bb
at ExThread.fillRequest()
Thread3 waits to acquire lock on synchronized code segment.

Will show up as Blocked or Waiting for Lock (on LockObj@0x849bb)

void waitForLock() {
// wait to acquire lock on lockObj
synchronized(lockObj) {
doSomething3();
}
//lock released
}

"ExecuteThread: '3‘
-- Blocked trying to get lock: LockObj@0x849bb
at ExThread.waitForLock()



Reducing Locks

Now we know how locks can lead to blocked threads and affect application performance, how to reduce locking in application?

There are multiple options:

  1. Try to use the concurrent.locks package which provide extended capabilities compared to synchronized regions or method calls for timed lock acquisition, fairness among threads etc. But use with care as not following the guidelines can lead to locks never getting released after acquiring it and lot more suffering.
  2. Avoid synchronized methods. Go with smaller synchronized regions whenever possible. Try to use synchronizing on a specific object (locking a room or a cabinet) that needs to be protected rather than a bigger parent object (compared to locking the entire building).
  3. Increase the number of resources, where access to them leads to locking. For example, when there are finite resources, access requires owner lock the resource and release once done. If the number of resources are too little (like JDBC Connections or some pool for user objects) compared to user threads requesting for the resource, there would be higher contention. Try to increase the number of resources.
  4. Try to cache resources if each call to create the resource requires synchronized calls. 
  5. Try to avoid the synchronized call entirely if possible by changing the logic of execution.
  6. Try to control the order of locking in cases of deadlocks. For example, every thread has to obtain LockA before obtaining LockB and not mix up the order of obtaining locks.
  7. If the owner of the lock has to wait for an event, then do a synchronization wait on the lock which would release the lock and put the owner itself on the blocked list for the lock automatically so other threads can obtain the lock and proceed. 

Summary
In this section, we went over basics of thread states and thread locking. In the next section, we will drill deeper into capturing and analyzing Thread Dumps with special look into WebLogic Application Server specific thread dumps.