06 — Common Production Pathologies

Every senior Java engineer has a small library of patterns they’ve seen fail in production. The specifics of the app vary — payments, ads, video, ML serving — but the shape of the failure repeats. Long GC pause under load. OutOfMemoryError: Direct buffer memory on a Netty service. Threads all parked on a JDBC connection pool. Container getting OOM-killed at 4 GB even though -Xmx is 3 GB.

This file catalogs the patterns you should recognize on sight, with the diagnostic path for each. When you’ve built this catalog, on-call stops being scary — you start noticing which pattern you’re looking at within the first few minutes.

1. The Six Species of OutOfMemoryError

Not all OOMs are the same. The message string tells you which region ran out and therefore where to look. Get this table into your head cold.

Message	What ran out	First move
Java heap space	Heap — objects	Heap dump, MAT dominator tree
GC overhead limit exceeded	Heap — GC spending >98 % CPU reclaiming <2 % memory	Same as above; the app is doomed but crawling
Metaspace	Class metadata (native)	jcmd GC.class_histogram; check for class-loader leak
Direct buffer memory	Off-heap ByteBuffers	Check Netty pools, DirectByteBuffer allocation sites
unable to create new native thread	OS thread limit or address space	ulimit -u, thread count, virtual threads?
requested N bytes for X. Out of swap space?	Malloc failure — native library	NMT summary; native leak or OS-level pressure

Two examples worth walking through:

Heap OOM in a caching service. Symptom: java.lang.OutOfMemoryError: Java heap space, hprof dumped. Open in MAT. Dominator tree shows a ConcurrentHashMap retaining 3.2 GB. Look at the entries: cached objects with no TTL, no eviction, and one entry per user request ID. That’s an unbounded cache. Fix: replace with Caffeine + a maximumSize or expireAfterWrite.

Metaspace OOM in an app server. Symptom: OutOfMemoryError: Metaspace after ~50 hot redeploys of a WAR. jcmd VM.native_memory summary shows Metaspace at 2 GB and growing. Class-loader leak — old versions of your classes are still reachable, usually because a thread pool, ThreadLocal, or JDBC driver registered a leak-preventing reference. Fix in the short term: -XX:MaxMetaspaceSize=512m + restart. Fix properly: find the retention path (there’s a leak-detection MAT plugin) and null the reference.

2. Long GC Pauses

You have a 200 ms SLO. Occasional 800 ms pauses ruin it. What causes them, in rough order of frequency:

1. Live set too big for the collector. G1 pause targets are hints, not guarantees. If your live set fills 80 % of Old, the mark phase is expensive and Mixed cycles get long. Fix: bigger heap, or ZGC on Java 21+.

2. Humongous allocations. A 20 MB byte[] skips Eden and lands in Old, potentially fragmenting and triggering earlier Mixed GCs. Fix: break the allocation up, or increase region size with -XX:G1HeapRegionSize.

3. String deduplication or class unloading pauses. Rare but real. Visible in GC logs as separate phases.

4. Safepoint stalls. Not GC per se, but manifests the same way. A thread is slow to reach the next safepoint (typically the back edge of a counted loop) and the whole world waits. Print with -Xlog:safepoint. Long safepoint sync times ≠ long GC, but users can’t tell the difference.

5. Full GC. Any Pause Full line in GC logs on G1 is bad. Investigate.

Diagnose with GC logs (-Xlog:gc*), GCViewer or GCEasy for graphs, and correlate with JFR to see what threads were doing during the pause.

3. Thread Starvation

Threads exist, they just can’t do work. Common shapes:

Connection pool exhaustion. HikariCP pool of 20, all in use, request threads blocked in getConnection() waiting for connectionTimeout (default 30 s). Thread dump shows N threads all parked at HikariPool.getConnection. Fix: increase pool size (carefully — DB has connection limits too), reduce query duration (slow queries hold connections), or add circuit breakers.

All threads doing I/O. Tomcat with 200 workers, all waiting on a downstream service that’s slow. Every incoming request queues, and eventually the accept queue fills. Fix (2024+ answer): switch that service to virtual threads. Then the “200 threads” ceiling isn’t a ceiling.

Uncaught exceptions killing pool threads. ScheduledExecutorService.scheduleAtFixedRate silently stops running future invocations if a scheduled task throws an uncaught exception (this exact trap is in the Phase 5 executor file). No log, no alarm, no metric — just quiet cessation. Wrap every runnable/callable in a try/catch that at minimum logs.

4. Connection Pool Exhaustion (Deep Dive)

The single most-common Java production incident, in most shops. Symptoms line up:

• Latency shoots up.

• Errors return SQLException: unable to obtain connection.

• Thread dump shows most workers parked at HikariPool.getConnection or com.zaxxer.hikari.pool.HikariPool.getConnection.

Root causes, ranked:

1. Slow query. One query starts taking 5 s (unindexed WHERE, SELECT * returning 1M rows, a lock wait). Connections stay held. Everything upstream backs up.

2. Connection leak. A code path doesn’t close() (or doesn’t use try-with-resources). Pool fills, never drains. Enable HikariCP’s leakDetectionThreshold=30000 — it logs stack traces of leaks.

3. Pool sized wrong. Too small = starvation under load. Too large = database dies. HikariCP’s rule of thumb: pool_size = (cores * 2) + effective_spindle_count. Usually 10–30 for OLTP.

4. DB-side deadlock or long transaction. App connections wait on the DB. Look at pg_stat_activity or SHOW PROCESSLIST.

Metrics you must have: pool active/idle/waiting, getConnection p99 latency, connection acquisition wait time. HikariCP publishes these via Micrometer for free.

5. Native Memory Growth (RSS > Heap)

The container is at 12 GB RSS. -Xmx is 4 GB. Nobody understands. This is one of the highest-signal skills you can carry to an study.

Native memory categories (from jcmd VM.native_memory summary):

Category	Common culprit
Java Heap	-Xmx. Bounded, boring.
Class (Metaspace)	Class-loader leak, hot redeploy
Thread	Stack size × thread count. 1000 platform threads × 1 MB = 1 GB.
Code	JIT-compiled code. Grows with codebase size + -XX:ReservedCodeCacheSize.
GC	GC’s own data structures (card tables, remembered sets). Grows with heap.
Compiler	C1/C2 scratch space. Usually small.
Symbols	String tables, class symbols.
Native Memory Tracking	The bookkeeping itself.
Internal	Everything else — direct buffers, JNI allocations

Enable NMT at startup: -XX:NativeMemoryTracking=summary. Cost: ~5 % memory overhead. Worth it in prod-adjacent envs.

Diagnosing “container OOM at 12 GB”:

jcmd <pid> VM.native_memory summary
jcmd <pid> VM.native_memory baseline
# ... wait for growth ...
jcmd <pid> VM.native_memory summary.diff

The diff shows which category grew. If Thread grew, count threads. If Class grew, class-loader leak. If Internal grew, look at direct buffers (Netty!) and JNI libraries.

6. Container-Aware JVM Settings

Since Java 10 (and improved in 11, 15, 17), the JVM reads cgroup limits and sizes itself accordingly — if you don’t override the defaults badly.

Recommended container flags:

-XX:MaxRAMPercentage=75.0      # leave 25 % of pod memory for non-heap: metaspace, threads, direct, code cache
-XX:InitialRAMPercentage=75.0  # same as -Xms=-Xmx, avoids resize pauses
-XX:MaxMetaspaceSize=512m      # cap explicit
-XX:MaxDirectMemorySize=1g     # cap explicit for Netty-heavy apps
-XX:+ExitOnOutOfMemoryError    # prefer crash + restart over zombie
-XX:+HeapDumpOnOutOfMemoryError
-XX:HeapDumpPath=/var/log/heapdumps/
-Xlog:gc*:file=/var/log/gc.log:time,uptime:filecount=10,filesize=100M

Why not -Xmx3g explicitly? You can, but percentage-based sizing scales when someone changes the pod’s memory limit in the deployment YAML without touching the JVM args. Explicit -Xmx gets stale.

Kubernetes gotcha: requests.memory vs limits.memory. The JVM sizes off limits, not requests. If requests is 2 GB and limits is 8 GB, JVM assumes 8 GB is available, allocates aggressively, and gets OOM-killed the moment the node is under memory pressure and enforces requests. Set requests ≈ limits for JVM pods, or accept the risk.

7. When the JIT Isn’t Helping

Rare but real: a service that runs well in staging (warm JVM) and terribly on new pods (cold JVM) for the first ~30 seconds after start. Options:

• App warmup: run synthetic traffic through hot paths before opening the pod to the load balancer. Spring Boot 3’s readiness probe + a warmup endpoint you hit before flipping ready.

• AppCDS (Application Class Data Sharing, -XX:SharedArchiveFile): pre-shares loaded class metadata across JVM instances. Cuts startup by 20–40 %.

• Class Data Sharing (dynamic): -XX:ArchiveClassesAtExit=app.jsa writes a shared archive on shutdown; next run loads it. JDK 13+.

• Coordinated Restore at Checkpoint (CRaC): snapshot a warmed-up JVM and restore it (JDK 21+, on Linux). Restore in <100 ms. Not yet mainstream but climbing.

• native-image: the nuclear option. Startup in 20 ms, no JIT, trade-offs from file 03.

8. A Playbook

When paged, in order:

1. Read the alert. What breached? Latency? Error rate? Memory? CPU?

2. Grab a thread dump (jcmd <pid> Thread.print). Answers “what are threads doing right now?”

3. Grab GC log tail. Answers “is GC the problem?”

4. Check heap + native memory (jcmd GC.heap_info, jcmd VM.native_memory summary). Answers “am I running out of something?”

5. Start a JFR recording if the app is still up (jcmd JFR.start duration=60s). Answers “what will this look like when I have time to analyze?”

6. If nothing conclusive, restart the process to unblock users, but not before you have the artifacts from steps 2–5. Restarting without artifacts is guaranteeing a repeat incident.

⚠️ What Most People Get Wrong

“Restart the pod, ticket closed.” The single worst on-call habit in Java shops. Every restart without artifacts throws away the only chance you had to fix the underlying cause. The alert will fire again in three days, on someone else’s on-call. Capture the thread dump, heap dump, and last 60 s of GC log before you kill the process. Even a mediocre artifact is infinitely more useful than “yeah, it happened again, we restarted it.”

“We’ll add more memory.” Sometimes correct, often expensive theater. If the app leaks, adding memory delays the failure by a proportional amount. If the app allocates too fast, adding memory means longer GC pauses. If the app is unbounded-caching, adding memory grows the cache to the new limit. Measure first, resize second — and only after the measurement supports it.

Recap

• Six species of OOM: Java heap, GC overhead, Metaspace, Direct buffer, unable to create native thread, Out of swap. Each maps to a different diagnostic path.

• Long GC pauses: live set too big, humongous allocations, safepoint stalls, or Full GC. Not always GC — sometimes safepoints.

• Thread starvation: connection pool exhaustion, downstream slow, or silent scheduled-task death.

• HikariCP leakDetectionThreshold catches connection leaks. Pool sizing rule: 2 × cores + spindles.

• NMT explains RSS > heap. Enable with -XX:NativeMemoryTracking=summary.

• Container flags: MaxRAMPercentage=75, explicit Metaspace + DirectMemory caps, HeapDumpOnOOM, ExitOnOOM.

• Capture artifacts before restarting. No exceptions.

---

Return to README.md · Previous: 05_benchmarking_with_jmh.md · Next: projects.md