04 — Diagnostic Tools

A senior Java engineer is defined less by the code they write than by what they can find out about a running JVM they didn’t write. Someone hands you a production PID and says “it’s slow, figure it out.” The tools in this file are how you answer.

You’ll build fluency in three layers: command-line inspection (jcmd, jstack, jmap, jstat), event recording (JFR + JMC), and profiling (Async Profiler, Eclipse MAT). Every one ships with modern OpenJDK or is a small download away. None of them cost money. All of them are boring to install and transformative once you know them.

1. jcmd — the Swiss Army Knife

Everything the older jstack/jmap/jinfo do, jcmd does too — and more, and more consistently. Use it first. Learn its shape:

jcmd                          # list running Java PIDs
jcmd <pid> help               # list commands available on that JVM
jcmd <pid> Thread.print       # thread dump (replaces jstack)
jcmd <pid> Thread.dump_to_file -format=json /tmp/threads.json   # Java 21+
jcmd <pid> GC.heap_dump /tmp/heap.hprof                         # heap dump (replaces jmap -dump)
jcmd <pid> GC.heap_info       # current heap sizes
jcmd <pid> GC.class_histogram # top N classes by instance count / bytes
jcmd <pid> VM.native_memory summary        # NMT — native memory tracking
jcmd <pid> VM.system_properties
jcmd <pid> VM.flags -all      # every VM flag, defaulted or set
jcmd <pid> JFR.start name=session1 duration=60s filename=/tmp/rec.jfr
jcmd <pid> JFR.stop  name=session1

NMT (Native Memory Tracking) is criminally underused. Start the JVM with -XX:NativeMemoryTracking=summary, then jcmd <pid> VM.native_memory summary shows you every category of native memory: heap, class, thread, code, GC, compiler, internal. This is how you diagnose “my -Xmx is 4G but the container is using 12G RSS.”

2. jstack — Thread Dumps

jstack <pid> dumps every thread’s stack. Still useful, still the shorthand. Use it when you want a quick one-shot to stdout:

jstack -l <pid> > threads.txt      # -l adds lock info (which threads hold which monitors)

On Linux/macOS you can also send SIGQUIT:

kill -3 <pid>                       # dumps to the JVM's stdout, wherever that is

Interpretation is covered in 05_concurrency_multithreading/06_debugging_concurrency.md. In this phase, thread dumps show up when diagnosing thread starvation, connection pool exhaustion, and deadlock.

3. jmap and Heap Dumps

jmap is the legacy heap tool; on modern JVMs jcmd GC.heap_dump is preferred. To capture:

jcmd <pid> GC.heap_dump /tmp/heap.hprof            # forces a Full GC first, by default
jcmd <pid> GC.heap_dump -all /tmp/heap.hprof       # includes unreachable objects

Do this on every prod pod: set -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/var/log/heapdumps/ at startup. When (not if) you hit an OOM, you get a .hprof file automatically. Without this flag, an OOM leaves you with a stack trace and nothing else — you cannot go back in time.

Reading Heap Dumps with Eclipse MAT

Eclipse Memory Analyzer (MAT, eclipse.dev/mat/) is the standard heap analyzer. Free, standalone, opens .hprof files up to tens of gigabytes.

The two views you’ll live in:

  • Histogram — every class, count, shallow size, retained size. Sort by retained size to find who’s really using memory.

  • Dominator Tree — shows the object that retains the most memory. If you remove this object, all its dominated children become garbage. This is how you find the leak’s root, not just its symptoms.

MAT’s Leak Suspects Report is genuinely good — run it on any dump and it will suggest one or two dominators worth investigating. It’s not always right but it’s a great starting point.

4. jstat — GC Metrics in a Terminal

jstat prints periodic GC and JIT statistics. Nice for a quick live view when you don’t want to attach a full profiler:

jstat -gc <pid> 1s               # every 1 second, all GC counters
jstat -gcutil <pid> 1s           # percentage utilization per region
jstat -gccause <pid> 1s          # includes cause of last GC

Interpret the columns from -gcutil:

Col

Meaning

S0, S1

Survivor 0 / 1 utilization (%)

E

Eden utilization (%)

O

Old utilization (%)

M

Metaspace utilization (%)

YGC / YGCT

Young GC count / total time (s)

FGC / FGCT

Full GC count / total time (s)

GCT

Total GC time

Watch E climb and reset (normal), O slowly climb (fine), O climb fast (leak or under-provisioned heap), FGC ever increase (bug).

5. Java Flight Recorder + Mission Control

JFR is the JVM’s built-in low-overhead event recorder. Under 1 % overhead in default mode. Since Java 11 it’s open-source and free in production.

Recording

# Start-time flag — always-on, rolling last hour to disk
-XX:StartFlightRecording=duration=1h,filename=/var/log/jfr/app.jfr,settings=profile

# Attach live to a running PID
jcmd <pid> JFR.start name=session1 duration=60s settings=profile filename=/tmp/rec.jfr
jcmd <pid> JFR.dump  name=session1 filename=/tmp/rec.jfr
jcmd <pid> JFR.stop  name=session1

Two default configurations ship with the JVM: default (very low overhead, always safe) and profile (more detailed, ~2 % overhead, use for investigation). You can also write custom .jfc configs to tune event detail.

Reading with JMC

JDK Mission Control (jdk.java.net/jmc/) is the free GUI viewer. Open the .jfr file and you get:

  • Method Profiling — sampled stack traces, flame graph view, top hot methods.

  • Memory — allocation rate, TLAB events, GC pauses graphed over time.

  • GC — pause distribution, before/after heap sizes, phase breakdowns.

  • Threads — lock contention, park/wait events, jdk.VirtualThreadPinned (Phase 5!).

  • I/O — socket read/write times, file I/O.

  • Exceptions — every thrown exception with rate and top-N by count.

The Automated Analysis page (JMC 8+) runs rules against the recording and produces prioritized findings: “heap live set is growing”, “91 % of GC pauses under 100 ms but 4 outliers over 500 ms”, etc. Start there.

6. Async Profiler — Where Modern Java Profiling Lives

Async Profiler (github.com/async-profiler/async-profiler) is the sampling profiler most senior Java engineers reach for first in 2024–26. It uses AsyncGetCallTrace (a not-quite-public HotSpot API) to sample stacks without the safepoint bias that plagued older tools like VisualVM’s sampler. It supports CPU, allocation, lock, wall-clock, and cache-miss profiling, and emits flame graphs directly.

Usage

# CPU flame graph, 30 seconds, output SVG
./profiler.sh -e cpu -d 30 -f /tmp/cpu.html <pid>

# Allocation profiling — who's allocating, by count and bytes
./profiler.sh -e alloc -d 30 -f /tmp/alloc.html <pid>

# Lock contention profiling
./profiler.sh -e lock -d 30 -f /tmp/lock.html <pid>

# Wall-clock profiling — shows blocking too, not just on-CPU
./profiler.sh -e wall -d 30 -f /tmp/wall.html <pid>

Each produces an interactive HTML flame graph. Widest bars = most time. Click to zoom. Search box highlights matching frames — use it to search for park, lock, read0, commitAndFlush etc.

Why Async Profiler > VisualVM / older tools

  • No safepoint bias. Traditional sampling profilers can only sample at safepoints, which distorts the profile toward methods with many safepoint polls. Async Profiler samples at any PC.

  • Allocation profiling is real. It samples TLAB allocation events, giving you “who allocates what” with tiny overhead. VisualVM can’t do this well.

  • Flame graphs are the default output. Same UX Brendan Gregg introduced for Linux perf. Once you use them you don’t go back.

VisualVM still exists, still ships free from Oracle. Use it for casual dev-desktop inspection. For production, Async Profiler + JFR.

7. jinfo — Flags at a Glance

jinfo -flags <pid>                  # every flag set on the running JVM
jinfo -flag +PrintGC <pid>          # toggle a manageable flag at runtime

Useful mostly to answer “what flags is this pod actually running with?” — which is often not what the deployment manifest claims, because environment scripts have a way of eating each other.

8. Choosing Your Tool: A Decision Table

Symptom

First tool

Second tool

“App is slow” (unspecified)

JFR 60s + JMC Automated Analysis

Async Profiler CPU

High CPU, no obvious hot spot

Async Profiler -e cpu

Async Profiler -e wall

Allocation-driven GC pressure

Async Profiler -e alloc

JFR TLABAllocation events

Lock contention suspected

Async Profiler -e lock

jstack at pain moment

OOM: heap

Heap dump + Eclipse MAT dominator tree

JFR live-set trend

OOM: Metaspace

jcmd GC.class_histogram diffed over time

NMT summary

OOM: direct memory

NMT summary; look for Netty pool

-XX:MaxDirectMemorySize cap

High RSS, low heap

NMT summary

Native memory profiling with Async Profiler -e itimer

Slow startup

JFR from -XX:StartFlightRecording at boot

-XX:+PrintCompilation

⚠️ What Most People Get Wrong

“I’ll attach VisualVM in production.” Please don’t. Older sampling profilers introduce measurable overhead and safepoint bias, and worse, they encourage you to eyeball a live view instead of recording and analyzing later. Record with JFR (always-on if you can afford it) and pull the file for offline analysis. Your future self, staring at a 3 a.m. incident recording six hours after the fact, will thank you.

“Heap dump when the pod is dying.” The pod already died; you don’t have time to run jcmd. Bake -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/var/log/heapdumps/ into the pod spec, mount the path to persistent storage or a sidecar that ships it to S3, and test the recovery path before you need it. “We had a heap dump but nobody could get it off the pod” is a real story from every shop.

Recap

  • jcmd first — it does everything jstack/jmap/jinfo do, plus JFR control and NMT.

  • Set -XX:+HeapDumpOnOutOfMemoryError in every production JVM.

  • JFR is the always-on recorder; JMC opens the file. settings=profile for investigation, default for always-on.

  • Async Profiler is the modern profiler: CPU, alloc, lock, wall-clock, flame-graph output, no safepoint bias.

  • Eclipse MAT for heap dumps — dominator tree + leak suspects report.

  • NMT (-XX:NativeMemoryTracking=summary + jcmd VM.native_memory) is the only way to explain “heap is fine, RSS is not.”

Return to README.md · Previous: 03_jit_and_optimization.md · Next: 05_benchmarking_with_jmh.md