Debugging Concurrency — Reading What the JVM Is Actually Doing

Writing concurrent code is hard. Debugging it is harder because most tools designed for single-threaded reasoning stop working: a debugger’s breakpoint changes the timing, println reorders itself, and “it worked when I stepped through it” is a warning sign, not a diagnosis.

This file is about the tools and habits that actually work — thread dumps, deadlock reports, race-detection frameworks, and controlled chaos. When you finish it, you should be able to look at a 2000-line jstack output at 3 AM and pinpoint what’s wrong within a few minutes.

---

1. Thread Dumps — Your First Weapon

A thread dump is a snapshot of every live thread’s stack. It tells you what each thread is doing right now.

Getting a dump

Command	Use case
jstack <pid>	Simple, prints to stdout. Available in JDK.
jcmd <pid> Thread.print	Same info, slightly richer. Preferred modern way.
jcmd <pid> Thread.dump_to_file -format=json dump.json	JSON format, machine-readable, includes virtual thread hierarchy. Java 21+.
kill -3 <pid>	Sends SIGQUIT; JVM prints a dump to its stdout. Works when jstack can’t attach.
Grab one every 5 seconds for 30 seconds	Compare across dumps to catch what’s actually stuck vs momentarily busy.

Reading a dump

Example entry:

"http-nio-8080-exec-42" #94 daemon prio=5 os_prio=0 cpu=1234.56ms
   java.lang.Thread.State: BLOCKED (on object monitor)
    at com.example.Service.transfer(Service.java:47)
    - waiting to lock <0x00000007a0e12340> (a com.example.Account)
    - locked <0x00000007a0e12350> (a com.example.Account)
    at com.example.Handler.handle(Handler.java:22)

Key signals:

• BLOCKED (on object monitor) — waiting to acquire a synchronized lock. If many threads BLOCKED on the same monitor address, that’s contention.

• WAITING (parking) on a LockSupport — waiting inside ReentrantLock, a queue, or a CompletableFuture. The next frames tell you which.

• TIMED_WAITING (sleeping) — Thread.sleep, wait(timeout). Usually benign unless it’s a hot path.

• RUNNABLE — executing Java code (or blocked in an OS call the JVM can’t see, e.g., some native I/O).

Look at the lock addresses. <0x00000007a0e12340> is a specific monitor. If thread T1 is waiting for that address and thread T2 is locked on it, T2 is blocking T1.

The deadlock section

At the bottom of a jstack dump, if there’s a cycle, the JVM prints an explicit deadlock report:

Found one Java-level deadlock:
=============================
"Thread-1":
  waiting to lock monitor 0x00007f8a... (object 0x00000007a0e12340, a Account),
  which is held by "Thread-2"
"Thread-2":
  waiting to lock monitor 0x00007f8a... (object 0x00000007a0e12350, a Account),
  which is held by "Thread-1"

This is the fastest debugging experience in Java. If you see it, you have the exact fix path: figure out why the two threads acquire in different orders, impose an ordering, done.

---

2. jcmd — The Diagnostic Swiss Army Knife

jcmd has replaced most legacy tools. Learn these commands.

jcmd <pid> help                           # list all commands
jcmd <pid> Thread.print                   # thread dump
jcmd <pid> GC.heap_info                   # heap layout
jcmd <pid> GC.heap_dump /tmp/heap.hprof   # heap dump
jcmd <pid> VM.system_properties           # -D props
jcmd <pid> VM.flags                       # all JVM flags in effect
jcmd <pid> VM.native_memory summary       # NMT if -XX:NativeMemoryTracking=summary
jcmd <pid> JFR.start duration=60s filename=/tmp/rec.jfr   # start a Flight Recording

jcmd is delivered with every JDK. Nothing to install. Runs in the target JVM’s process context via the Attach API. Master this before anything else.

---

3. Deadlock Prevention Techniques (Runtime)

Re-cap from file 02, now with runtime detection:

1. Impose a global lock order — acquire by increasing ID or System.identityHashCode.

2. Prefer tryLock with timeout — replace deadlock (unrecoverable) with livelock (recoverable) or a fail-fast error.

3. Watchdog: in production, sample thread dumps periodically and alert if any pair of threads has been BLOCKED > threshold. Both Datadog and New Relic have this out of the box.

For debugging a suspected deadlock right now: run the process, wait for the hang, take a jstack, read the deadlock section. Done.

---

4. Race Conditions — The Hard Class

A deadlock stops the program. A race condition silently produces wrong answers, sometimes. Debugging races is harder because they’re non-reproducible on demand.

4.1 jcstress — The Only Correct Tool

Java Concurrency Stress Tests is the framework that Aleksey Shipilëv (OpenJDK performance lead) built to test the JMM itself. It runs your code in tight loops on many threads for millions of iterations and enumerates the actual observed outcomes, so you can catch memory-model violations that your ad-hoc tests will never trigger.

@JCStressTest
@Outcome(id = "1, 1", expect = ACCEPTABLE, desc = "Both threads saw the update.")
@Outcome(id = "0, 0", expect = ACCEPTABLE_INTERESTING, desc = "Reordering visible.")
@State
public class RaceTest {
    int x, y;

    @Actor public void thread1(II_Result r) { x = 1; r.r1 = y; }
    @Actor public void thread2(II_Result r) { y = 1; r.r2 = x; }
}

Run it, and jcstress will tell you exactly which outcomes your code produces — including the interesting ones that only happen once in ten million. This is how the JDK itself validates concurrent code. If you’re writing a lock-free algorithm, you write a jcstress test alongside it.

4.2 Chaos with Thread.yield()

A cheaper first-pass technique for tests: sprinkle Thread.yield() at suspected interleaving points. It hints to the scheduler that another thread may run. Not a proof of correctness, but it dramatically increases the odds of hitting a race in a unit test.

public void increment() {
    int c = counter;
    Thread.yield();          // encourage interleaving in tests
    counter = c + 1;
}

Remove before shipping. This is for CI stress runs, not production.

4.3 TSAN — Concepts, Not the Tool

Go and Rust have thread sanitizers (TSAN) that instrument every load/store and detect races at runtime with vector clocks. Java doesn’t ship one. The philosophical replacement is jcstress plus the JMM: prove correctness by argument, verify by stress test.

---

5. Reproducing a Race — The Workflow

You have a report: “user occasionally sees stale data”. Nothing reproduces. Here’s the drill:

1. Read the code. Identify all shared mutable state. If you cannot list every field that two threads touch, stop and re-read.

2. List the happens-before edges. For each pair of read/write, ask: is there a lock, a volatile, a queue handoff between them? If not, mark it suspect.

3. Write a jcstress test targeting that pair. Run for a few minutes with many actors. Watch for ACCEPTABLE_INTERESTING outcomes.

4. Fix by adding an edge: a lock, volatile, immutable snapshot, or replacement with a concurrent collection.

5. Rerun jcstress. The interesting outcome should be gone.

6. Regression-test with a chaos test (many threads, Thread.yield sprinkled) as CI protection.

Do not try to reproduce a race with a main method that loops 1000 times. That’s how one-in-a-million bugs stay one-in-a-million.

---

6. Beyond Java-Level: When the JVM Is the Problem

Some symptoms look like concurrency bugs but are actually GC, JIT, or kernel issues:

• Sudden 200 ms hiccups: possibly a GC pause. Check -Xlog:gc* output. Covered in Phase 06.

• Thread stuck in runnable but no CPU used: possibly a native syscall the JVM can’t introspect. Use perf, strace, or Async Profiler (Phase 06).

• All threads suddenly go RUNNABLE with weird stacks: possibly a JIT deoptimization storm. Add -XX:+PrintCompilation.

When your Java-level debugging bottoms out, the JVM-level tools in Phase 06 are the next layer.

---

7. Cheat Sheet

Symptom	First thing to do
Process hangs	3 × jstack five seconds apart; look for deadlock section
High CPU, no progress	jstack; look for a hot loop or lock convoy
Occasional wrong result	jcstress test on the suspect pair
p99 latency spikes	JFR recording; check jdk.VirtualThreadPinned, GC pauses
“Works locally, fails in prod”	Different scheduler + different load. Trust jcstress over anecdote.

---

⚠️ What Most People Get Wrong

They “debug” concurrency by adding println and staring at the log. Print statements are internally synchronized, so they linearize execution and hide the very race you’re trying to catch. If you must instrument, use a lock-free ring buffer or JFR events — not System.out.

They also treat a passing test suite as proof. It is not proof. A single passing run of a race-condition test proves that on that machine, that JIT compile, that scheduler decision, the code didn’t fail. Only jcstress produces evidence approaching proof. Everything else is anecdote.

And the study classic: they cannot read a jstack dump. Practice on real dumps. Print one from your own project, right now, and try to name what each thread is doing. If you can’t, that’s your first hour of homework for this file.

---

Return to README.md · Next: projects.md