We will also discuss how it helps to pinpoint the issues and some of the analyzer you can use.
What is Thread?
A process is a computer program which is loaded into the computer’s memory and is under execution. It can be executed by a processor or a set of processors. A process is described in memory with important information such as variable stores, file handles, the program counter, registers, and, signals, and so on. A process can consist of many lightweight processes called threads. This helps to achieve parallelism wherein a process is divided into multiple threads. This results in better performance. All the threads within a process share the same memory space and are dependent on each other.
Thread Dumps
When the process is executing, we can detect the current state of execution of the threads in the process using thread dumps. A thread Dump contains a snapshot of all the threads active at a particular point during the execution of a program. It contains all relevant information about the thread and its current state. A modern application today involves multiple numbers of threads. Each thread requires certain resources, performs certain activities related to the process. This can boost the performance of an application as threads can utilize available CPU cores. But there’s are trade-offs, e.g., sometimes multiple threads may not coordinate well with each other and a deadlock situation may arise. So, if something goes wrong, we can use thread dumps to inspect the state of our threads.
Thread dump in Java
A JVM thread Dump is a listing of the state of all threads that are part of the process at that particular point of time. It contains information about the thread’s stack, presented as a stack trace. As it is written in plaintext, the contents can be saved for reviewing later. Analysis of thread dumps can help in
Optimizing JVM performance Optimizing application performance Diagnosing problems, e.g. a deadlock, thread contention, etc.
Generation of Thread Dumps
There are many ways to generate thread dumps. Below are some JVM based tools and can be executed from the command line/terminal (CLI tools) or the /bin (GUI tools) directory of the installation folder of Java. Let’s explore them.
#1. jStack
The simplest way to generate a thread dump is by using jStack. jStack ships with JVM and can be used from the command line. Here, we need the PID of the process for which we want to generate the thread dump. To get PID we can use jps command as shown below. jps -l jps lists down all java process ids.
On Windows
On Linux
As we can see here, we get a list of all running java processes. It contains the local VM id for the running java process and the name of the application in columns one and two respectively. Now, to generate the thread dump, we use the jStack program with –l flag which creates a long listed output of the dump. We can also pipe the output to some text file of our choice.
jstack -l 26680
#2. jvisualvm
Jvisualvm is a GUI tool that helps us troubleshoot, monitor, and profile Java applications. It also comes with JVM and can be launched from the /bin directory of our java installation. It is very intuitive and easy to use. Among other options, it also allows us to capture thread dump for a particular process. To view the thread dump for a particular process, we can right-click on the program and select Thread Dump from the context menu.
#3. jcmd
JCMD is a command-line utility that ships with the JDK and are used to send diagnostic command requests to the JVM. It however works only on the local machine where the Java application is running. It can be used to control Java Flight Recordings, diagnose and troubleshoot JVM and Java applications. We can use the Thread.print command of jcmd to get a list of thread dumps for a particular process specified by the PID. Below is an example of how we can use jcmd. jcmd 28036 Thread.print
#4. JMC
JMC stands for Java Mission Control. It is an open-source GUI tool that ships with JDK and is used to collect and analyze java application data. It can be launched from the /bin folder of our Java installation. Java administrators and developers use the tool to gather detailed low-level information about the JVM’s and application’s behaviors. It enables detailed and efficient analysis of data collected by Java Flight Recorder. On launching jmc, we can see list of java process which is running on the local machine. A remote connection is also possible. On a particular process, we can right click and choose Start Flight Recording and then check the thread dumps in the Threads tab.
#5. jconsole
jconsole is a Java Management Extension tool used for complaint management and monitoring. It also has a set of predefined operations on the JMX agent which the user can perform. It enables the user in detecting and analyzing stack trace of a live program. It can be launched from the /bin folder of our Java installation. Using the jconsole GUI tool we can inspect each thread’s stack trace when we connect it to a running java process. Then, in the Thread tab, we can see the name of all running threads. To detect a deadlock, we can click on the Detect Deadlock in the bottom right of the window. If a deadlock is detected it will appear in a new tab otherwise a No Deadlock Detected will be displayed.
#6. ThreadMxBean
ThreadMXBean is the interface for the management of the thread system of the Java virtual machine belonging to java.lang.Management package. It is mainly used to detect the threads which have entered a deadlock situation and get details about them. We can use the ThreadMxBean interface to programmatically capture the thread dump. getThreadMXBean() method of ManagementFactory is used to get an instance of the ThreadMXBean interface. It returns the number of both daemon and non-daemon live threads. ManagementFactory is a factory class for getting the managed beans for the Java platform.
Manual Analysis of Thread Dumps
Analysis of thread dumps can be very useful in pinpointing issues in multithreaded processes. Issues such as deadlocks, lock contention, and excess CPU utilization by individual thread dumps can be resolved by visualizing the states of individual thread dumps. Maximum throughput from the application can be achieved by rectifying the status of each thread after analyzing the thread dump. For instance, let’s say, a process is using up a lot of CPU, we can find out if any thread is using the CPU the most. If there’s any such thread, we convert its LWP number to a hexadecimal number. Then from the thread dump, we can find the thread with nid equal to the previously obtained hexadecimal number. Using the stack trace of the thread we can pinpoint the issue. Let’s find out the process id of the thread using the below command. ps -mo pid,lwp,stime,time,cpu -C java Let’s have a look at below chunk of thread dump. To get thread dump for process 26680, use jstack -l 26680 Now, Let’s see what are the things we can explore using thread dumps. If we observe the thread dump, we can see a lot of content, which can be overwhelming. However, if we take one step at a time, it can be fairly simple to understand. Let’s understand the first line 2020-06-27 09:01:29Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.221-b11 mixed mode): The above displays the time the dump was generated, and information about the JVM which was used. Next, in the end, we can see the list of threads, the first among them is our ReferenceHandler thread.
Analyzing Blocked Threads
If we analyze the below thread dump logs, we can find that it has detected threads with BLOCKED status which makes the performance of an application very slow. So, if we can find the BLOCKED threads, we can try to extract the threads related to the locks that the threads are trying to obtain. Analysis of the stack trace form the thread currently holding the lock can help in solving the problem.
Analyzing Deadlocked Thread
Another very commonly used application of thread dumps is the detection of deadlocks. The detection and solution of deadlocks can be a lot easier if we analyze the thread dumps. So, none of the threads can continue execution, and this results in a deadlock situation and ends in the application getting stuck. If dreadlocks are present, then the final section of the thread dump will print out the information regarding the deadlock as follows. Here we can see the deadlock information in a fairly human-readable format. Other than this, if we sum up all the above chunk of thread dump together, then it states the below information.
Reference handler is the human-readable name of the thread. #2 is the thread’s unique id. daemon denotes if the thread is a daemon thread. The numeric priority of the thread is given by prio=10. The current status of the thread is denoted by waiting on condition. Then we see the stack trace, which includes the locking information.
Thread Dumps Analyzers
Besides manual analysis, there are numerous tools available for analyzing thread dumps, both online and offline. Below are some of the listed tools, which we can use based on the requirements. First, let’s explore online tools.
#1. Fast thread
Fast Thread is the DevOps engineer’s favorite thread dump analysis tool to troubleshoot complex production problems. This is an online Java thread dump analyzer, We can upload the thread dump as a file or we can directly copy and paste the thread dump. Depending on the size, it will analyze the thread dump and displays the information as shown in the screenshot. Features
Troubleshoot JVM crashes, slowdowns, memory leaks, freezes, CPU Spikes Instant RCA (don’t wait for Vendors) Intuitive Dashboard REST API support Machine Learning
#2. Spotify Thread Dump Analyzer
The Spotify Thread Dump Analyzer is licensed under version 2.0 of the Apache license. It is an online tool and accepts the thread dump as a file or we can directly copy and paste the thread dump. Depending on the size, it will analyze the thread dump and displays the information as shown in the screenshot.
#3. Jstack review
Jstack.review analyzes java thread dumps from within the browser. This page is the client-side only.
#4. Site 24×7
This tool is a prerequisite for detecting faulty threads degrading Java Virtual Machine(JVM) performance. Issues such as deadlocks, lock contention, and excess CPU utilization by individual thread dumps can be resolved by visualizing the states of individual thread dumps. Maximum throughput from the app can be achieved by rectifying the status of each thread provided by the tool. Now, let’s explore offline tools.
#1. JProfiler
JProfiler is one of the most popular thread dump analyzers among Java developers. JProfiler’s intuitive UI helps you resolve performance bottlenecks, pin down memory leaks, and understand threading issues. JProfiler supports profiling on the following platforms:
Windows macOS Linux FreeBSD Solaris AIX HP-UX
Below are some features that make JProfiler the top choice for profiling our applications on the JVM. Features
Supports database profiling for JDBC, JPA, and NoSQL Support for Java enterprise edition is also available Presents high-level information about RMI calls Stellar analysis of memory leaks Extensive QA capabilities The integrated thread profiler is tightly integrated with the CPU profiling views. Support for platforms, IDE’s, and application servers.
#2. IBM TMDA
IBM Thread and Monitor Dump Analyzer for Java (TMDA) is a tool that allows identification of hangs, deadlocks, resource contention, and bottlenecks in Java thread dumps. It is an IBM product but the TMDA tool is provided as without any warranty or support; however, they try to fix and enhance the tool over time.
#3. ManageEngine
ManageEngine applications manager can help to monitor JVM Heap and Non-Heap memory. We can even configure thresholds and be alerted by email, SMS, etc, and ensure a Java application is tuned well.
#4. YourKit
YourKit consists of the below products called it as a Kit.
Java Profiler – Fully featured low overhead profiler for Java EE and Java SE platforms. YouMonitor – Performance monitoring and profiling of Jenkins, TeamCity, Gradle, Maven, Ant, JUnit, and TestNG. .NET Profiler – Easy to use performance and memory profiler for .NET framework.
Conclusion
Now you know, how thread dumps are useful in understanding and diagnosing problems in multithreaded applications. With proper knowledge, regarding the thread dumps – their structure, the information contained in them, and so on – we can utilize them to identify the causes of the problems quickly.
