Using JVMTOP and JVMMON to analyze Tomcat CPU bottlenecks

Troubleshooting CPU bottlenecks on the Apache Tomcat application server can be challenging. Since Apache Tomcat is running as a Java Virtual Machine, understanding which JVM threads are consuming the overall Tomcat CPU can be a bit of a black box since this information is not easily accessible in regular monitoring tools. Recently, I have discovered a brilliant new tool called JVMTOP which provides direct insight into the performance of threads running on a Java Virtual Machine. Additionally, JVMTOP also shows in real-time which threads are blocked and also the id of the thread it is currently waiting on. In this article, I will provide a demonstration on how this tool works and what information you need to collect once you have identified the problematic Java thread(s) so that SAP can perform a deeper root cause analysis.

Identify the CPU bound thread using JVMTOP

I have made a slight modification to the JVMTOP application (updated the exectuable jar manifest for ease of use) and created a new batch file to launch the application. Download the version linked from this article when following this tutorial. Note, this tutorial was tested only against Apache Tomcat running SAPJVM

Download jvmtop.zip
Extract the files from jvmtop.zip to <INSTALL_DIR>\SAP BusinessObjects Enterprise XI 4.0\win64_x64\sapjvm\bin on your Apache Tomcat node
Open a command prompt and browse to <INSTALL_DIR>\SAP BusinessObjects Enterprise XI 4.0\win64_x64\sapjvm\bin
Open task manager, click view --> select columns, select PID (Process Identifier) and determine the process id of tomcat.exe
In the command prompt, execute the command: runJVMTOP.bat <PID>. For example:

JVMTOP will connect to the SAPJVM and list the top 10 Java threads which are consuming the most CPU. Notice, it will continue to stay connected and the values are refreshed automatically in real-time similar to the Unix version of top

COLUMN	DESCRIPTION
TID	JVM thread ID
NAME	Name of JVM thread
STATE	The state that the thread is currently in
CPU	Current CPU utilization (in ratio to available cpu time on all processors)
TOTALCPU	CPU utilization (in ratio to process cpu consumption) since the thread is alive
BLOCKEDBY	If this thread is blocked, this column shows the thread id which is blocking this thread

In the above example, we can easily identify that the thread creating the CPU bottleneck is http-8080-5 with thread id 48

Create thread dumps using JVMMON

Now that we have identified the offending thread, we need to generate a thread dump so that we can analyze the call stack of the code currently being executed by this thread. SAPJVM includes a utility called JVMMON which makes creating thread dumps very easy to do without a lot of additional configuration.

Open a command prompt and browse to <INSTALL_DIR>\SAP BusinessObjects Enterprise XI 4.0\win64_x64\sapjvm\bin
Execute the file jvmmon.exe and it will print out a list of running JVM processes and their corresponding process id (PID)
Enter the process id of the Apache Tomcat Server JVM and press enter
Enter the command “print to file” and when prompted for “file=” enter a path and file name for the first thread dump. NOTE: You must first create the folder structure where you intend to save the file.
Next, enter the command "print stacktrace" and at the same time, take a screenshot of the current JVMTOP output.

The threads shown in JVMTOP will match the thread details shown in the thread dump and then deeper root cause analysis can be performed by SAP
Repeat the thread dump / JVMTOP screenshot collection while the CPU is bottlenecked (at an interval of five minutes between each collection). The more iterations of this information that is collected, the easier it will be to diagnose the bottleneck
Exit JVMTOP by typing CTRL-C and choose (Y) to end the batch job. Use the command exit to exit from JVMMON