Valuable Tools For Diagnostics Gathering and Troubleshooting

Introduction The Oracle A-Team is often asked to help customers identify a myriad of JVM and SOA application issues.  Without fail, the customer will be asked for data regarding their application.  This is not application data, but rather data about the running application from the JVM’s perspective. The data we ask for normally includes Java […]

ThreadLogic version 0.95 available

  ThreadLogic version 0.95 is available for public download now. Additional Features Biggest change is support for externalizing the Advisories and group definitions. Users can use the pre-defined AdvisoryMap.xml to come up with custom …

Introducing ThreadLogic

Eric Gross and Sabha Parameswaran, from Oracle FMW Architects Team (The A-team) are happy to introduce ThreadLogic, an open source Thread Dump Analyzer tool, to Middleware community.Motivation behind ThreadLogic  The current set of TDA tools (Sam…

Analyzing Thread Dumps in Middleware – Part 2

This posting is the second section in the series Analyzing Thread Dumps in Middleware

This section details with when, how to capture and analyze thread dumps with special focus on WebLogic Application Server related thread dumps. Subsequent sections will deal with more real-world samples and tools for automatic analysis of Thread Dumps.

The Diagnosis

Everyone must have gone through periodic health checkups. As a starting point, Doctors always order for a blood test. Why this emphasis on blood test? Can’t they just go by the patient’s feedback or concerns? What is special about blood test?

Blood Test help via:

  • Capturing a snapshot of the overall health of the body (cells/antibodies/…)
  • Detecting Abnormalities (low/high hormones, elevated blood sugar levels)
  • Identifying and start focusing  on the problem areas with prevention and treatment.

The Thread Dump is the equivalent of a blood test for a JVM. It is a state dump of the running JVM and all its parts that are executing at that moment in time.

  • It can help identify current execution patterns and help focus attention on relevant parts
    • There might be 100s of components and layers but difficult to identify what is getting invoked the most and where to focus attention. A thread dump taken during load will highlight the most executed code paths.
  • Dependencies and bottlenecks in code and application design
  • Show pointers for enhancements and optimizations

When to capture Thread Dumps

Under what conditions should we capture thread dumps? Anytime or specific times? Capturing thread dumps are ideal for following conditions:

  1. To understand what is actively executed in the system when under load
  2. When system is sluggish or slow or hangs
  • Java Virtual Process running but Server or App itself not responding
  • Pages take forever to load
  • Response time increases
  • Application and or Server behavior erratic
  • Just to observe hot spots (frequently executed code) under load
  • For Performance tuning
  • For Dependency analysis
  • For bottleneck identification and resolution
  • To capture snapshot of running state in server

 

The cost associated with capturing thread dumps is close to near zero; Java Profilers and other tools can consume anywhere from 5 to 20% overhead while a thread dump is more of  a snapshot of threads which requires no real heavy weight processing on part of the JVM. There can be some cost only if there are too many interrupts to the JVM within real short intervals (like dumps forever every second or so).

How to capture Thread Dumps

There are various options to capture thread dumps. JVM vendor tools allow capture and gather of thread dumps. Operating System interrupt signals to the process can also be used in generating thread dumps.

Sun Hotspot’s jstack tool (under JAVA_HOME or JRE Home/bin) can generate thread dumps given the jvm process id. Similarly, jrcmd (from JRockit Mission Control) can generate thread dumps given a jvm process id and using the print_threads command. Both require to be run on the same operating system as the jvm process and both dump the output to the user shell.

In Unix, kill -3 <PID> or kill -SIGQUIT<PID> will generate thread dumps on the JVM process. But the output of the thread dump will go to the STDERR. Unless the STDERR was redirected to a log file, the thread dump cannot be saved when issued via kill signals. Ensure the process is always started and STDERR is redirected to a log (best practice to also have STDOUT redirected to same log file as STDERR).

In Windows, Ctrl-Break to the JVM process running in foreground can generate thread dumps. The primary limitation is the process needs to be running in the shell. Process Explorer in windows can also help in generating thread dumps but its much more problematic to get all the thread stacks in one shot. One has to wade through each thread and gets its stack. Another thing to keep in mind is, JVM might ignore interrupts if they were started with -Xrs flag.

WebLogic Application Server allows capture of thread dumps via additional options:

    1. WLS Admin Console can generate thread dumps. Go to the Server Instance -> Monitoring -> Threads -> Dump Threads option.
    2. Using weblogic’s WLST (WebLogic Scripting Tool) to issue Thread dumps.