Projects — Phase 06

Three projects. Together they take about 30–40 focused hours. Each one is designed to force a specific senior-level competency into your fingers: profiling a service you didn’t write, defending microbenchmark numbers with JMH, and staring an OOM in the face until it explains itself. Ship all three publicly on GitHub — they are portfolio-grade.

Acceptance criteria for the whole phase: you have real artifacts (flame graphs, JMH result tables, heap dumps, GC logs) checked into each repo, and a README.md that walks a reviewer through the story. study partners can smell copy-paste; make yours the real thing.

Project A: Profile the Slow Service (12–16 h)

You’ll be handed a deliberately slow HTTP service (build one, or fork a small Spring Boot demo you don’t already know cold) and asked to make it faster by measurement, not guesswork. This is the exact study scenario for principal engineer roles.

The service

Build a Spring Boot 3.2+ service with the following intentional pathologies. Keep it small (one controller, ~200 LOC) so the exercise is finding the bugs, not reading the codebase.

  1. A /report endpoint that fetches user data from an in-memory HashMap, formats it via String.format in a loop, and returns a JSON response.

  2. Populated with 100k users at startup.

  3. Deliberately slow paths:

    • String concatenation with + inside a loop building each row.

    • A synchronized block around an unnecessary read.

    • A regex compiled inside the hot path instead of a static Pattern.

    • A stream().collect(Collectors.toList()) where a raw for would allocate half as much.

The exercise

  1. Load-test the baseline with k6 or wrk at 200 concurrent users, 60 s. Record p50/p95/p99, throughput, CPU %, allocation rate.

  2. Attach Async Profiler in CPU mode for 30 s under load. Save the flame graph as flame-cpu-baseline.html.

  3. Attach Async Profiler in -e alloc mode. Save flame-alloc-baseline.html.

  4. Start a 60 s JFR recording (jcmd JFR.start settings=profile duration=60s). Save baseline.jfr. Open in JMC, screenshot the Automated Analysis page.

  5. From the three artifacts, identify the top 3 bottlenecks. For each, write one paragraph: what it is, how the artifact showed it, expected fix, expected magnitude of improvement.

  6. Apply the three fixes.

  7. Re-run all four measurements. Save flame-cpu-after.html, flame-alloc-after.html, after.jfr.

  8. Publish a WRITEUP.md in the repo: before/after latency table, side-by-side flame-graph screenshots, one paragraph per fix. This is your artifact.

Acceptance criteria

  • p99 latency reduced by at least 3× from baseline.

  • Allocation rate reduced by at least 2× (visible in -prof gc and Async Profiler alloc flame graph).

  • Every claim in WRITEUP.md has a screenshot or table backing it. No unsupported numbers.

  • Both .jfr files committed to the repo (or a release attachment).

  • One-command reproduction: ./gradlew bootRun + a documented k6 run load.js invocation.

Why this project matters

study partners at senior levels routinely ask “tell me about a time you profiled and improved a service.” If your answer includes flame graphs, JFR recordings, and before/after numbers, you skip a level. If your answer is “we increased the heap”, you fail the round.

Project B: The Five JMH Benchmarks (8–12 h)

Write, run, and interpret five JMH benchmarks. This is where you build defensive credibility — the ability to disprove wrong performance claims, including your own.

The five benchmarks

All under one JMH submodule. Each with @Fork(value=2), @Warmup(5, time=1), @Measurement(10, time=1), appropriate @BenchmarkMode.

  1. String concat vs StringBuilder across @Param({"5", "50", "500", "5000"}) sizes. Expected: they tie at 5, StringBuilder wins by orders of magnitude at 5000. This is your DCE-defeating warm-up problem.

  2. ArrayList vs LinkedList iteration. Iterate a pre-filled list of 100k Integers, sum the elements. Two variants each: index-based for loop and enhanced-for. Expected: ArrayList wins across the board, and index-based on LinkedList is catastrophically slow (get(i) is O(n)).

  3. HashMap vs TreeMap lookup. Pre-fill each with 10k keys. Benchmark get() of a random hit key. Expected: HashMap ~2–3× faster than TreeMap for random access; report both averages and 99th percentile via SampleTime.

  4. synchronized vs ReentrantLock under contention. A shared counter incremented by N threads (use @Threads(8) and @State(Scope.Benchmark)). Compare synchronized block, ReentrantLock, AtomicLong, LongAdder. Expected: LongAdder demolishes the others under contention; synchronized and ReentrantLock are close on Java 21+.

  5. Virtual thread vs platform thread task overhead. Benchmark the cost of executor.submit(() -> {}) (do-nothing task, Blackhole.consume inside) for newFixedThreadPool(200), newVirtualThreadPerTaskExecutor(), and newCachedThreadPool(). Expected: virtual thread creation is ~1–2 μs and doesn’t degrade with load; platform pool submit is a queue push (~100 ns) but capped at 200 concurrent tasks.

Acceptance criteria

  • Every benchmark uses Blackhole.consume(...) or returns a value. No void benchmarks with unused locals.

  • -prof gc reported for each benchmark. Allocation rate (gc.alloc.rate.norm, bytes/op) documented per variant.

  • Each benchmark’s results published as a Markdown table in RESULTS.md, with Score ± Error.

  • For each benchmark, one paragraph interpreting the result: is the difference statistically significant (do error bars overlap), and does the direction match your prior hypothesis? If it doesn’t, that’s more interesting — dig in and explain.

  • JDK version, JMH version, and hardware documented at the top of RESULTS.md.

Why this project matters

After this, you can walk into any team’s codebase, spot a “we cache this for performance” comment, and settle the argument in under an hour with a proper JMH benchmark. That skill is rare and highly leveraged.

Project C: Three OOMs You Caused on Purpose (6–8 h)

Deliberately cause and then fix three different kinds of OutOfMemoryError in a small, controlled repo. Ship the heap dumps and GC logs alongside the code. study partners love this exercise — it proves you understand the JVM’s memory categories deeply enough to engineer their failure.

The three OOMs

OOM 1 — Java heap space. Write a program that adds byte[] entries to a List<byte[]> in a loop, never releasing references. Run with -Xmx256m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=./dumps. Capture: the stack trace, the .hprof file, screenshot of MAT’s dominator tree showing the List as the retainer.

OOM 2 — Metaspace. Write a program that dynamically creates classes in a loop using ByteBuddy or a URLClassLoader with unique class names, never releasing the loader. Run with -XX:MaxMetaspaceSize=64m -XX:+HeapDumpOnOutOfMemoryError. Capture: the stack trace, jcmd GC.class_histogram output at 50 % Metaspace showing the class count growing, then the final trace when Metaspace fills.

OOM 3 — Direct buffer memory. Write a program that allocates ByteBuffer.allocateDirect(10 * 1024 * 1024) in a loop into a list, never releasing. Run with -XX:MaxDirectMemorySize=128m. Capture: the stack trace showing OutOfMemoryError: Direct buffer memory and NMT summary showing Internal / Direct Buffer category filling.

The three fixes

For each OOM, implement the fix and verify. Document what the fix is and why it works.

  1. OOM 1 fix. Replace the unbounded List with a Caffeine cache: Caffeine.newBuilder().maximumWeight(200_000_000).weigher((k, v) -> v.length).build(). Verify: heap stays below -Xmx256m indefinitely.

  2. OOM 2 fix. Close/dereference the class loader when done. Verify: GC.class_histogram shows class count stable across iterations. (Bonus: understand why the classloader must be unreachable — a static field holding a reference to any of its classes pins the whole loader.)

  3. OOM 3 fix. Explicitly release direct buffers, either by Cleaner, by reusing a pool via Netty’s PooledByteBufAllocator, or by not using direct buffers at all. Verify: NMT summary shows Internal/Direct stable.

Acceptance criteria

  • ./run.sh oom1, ./run.sh oom2, ./run.sh oom3 reliably reproduce each OOM within 30 s.

  • Each .hprof (or its equivalent for Metaspace/Direct — heap dumps aren’t the diagnostic for those) is committed or attached as a release.

  • ./run.sh fix1|fix2|fix3 variants run for at least 5 minutes without OOM.

  • WRITEUP.md documents each OOM’s exact error message, the region that ran out, the tool used to observe the growth, and why the fix works.

Why this project matters

Most engineers only ever see the OOMs their own code accidentally causes. Deliberately engineering three different species trains a kind of memory-anatomy reflex: you see the error message and instantly know which region, which tools, which categories of fix. That reflex is what a staff engineer brings to an incident.

Delivery Checklist

  • All three repos public on GitHub with clear README.md.

  • Artifacts committed or attached: flame graphs, JFR recordings, JMH RESULTS.md, heap dumps.

  • Every performance claim in every writeup has a screenshot or table backing it.

  • JDK 21+ used throughout. Note the exact JDK vendor + version.

  • Linked from your main portfolio index (Phase 12).

Return to README.md · Previous: 06_common_production_pathologies.md