Phase 05 Projects — Prove You Can Ship Concurrent Code

Reading about concurrency is the easy part. Shipping something that survives 10k concurrent requests and doesn’t deadlock at 3 AM is the part that gets you the offer. Three projects. Each one forces a different muscle group.

Project A: Thread-Safe Rate Limiter

Time estimate: 10-14 hours

Problem Statement

Build a rate limiter as a Java library with two algorithms — token bucket and sliding window log — that is safe under high concurrency, correct under contention, and observable in production.

Requirements

  • API: boolean tryAcquire(String key, int permits) and long acquireBlocking(String key, int permits) (returns wait time in nanos).

  • Configurable rate per key: e.g., “user-42 is allowed 100 requests/sec, burst 200.”

  • Two implementations behind the same interface:

    1. Token bucket — refills continuously, allows bursts up to capacity.

    2. Sliding window log — exact count of requests in the last N milliseconds.

  • Per-key state stored in ConcurrentHashMap<String, State> with computeIfAbsent.

  • Metrics: total permits granted, total rejections, current in-flight count, hot keys.

Technical Constraints

  • Must use: AtomicLong or LongAdder for counters, ConcurrentHashMap, ReentrantLock where locking is needed (not synchronized — you’re forward-compatible with virtual threads), System.nanoTime() for time (never currentTimeMillis — it goes backwards on NTP adjustments).

  • Must NOT use: Guava’s RateLimiter, Resilience4j, or any external limiter library. You’re building it.

  • Test with: at least 100 producer threads on a CountDownLatch gate, driving 1,000,000 total attempts. Verify: (a) the total granted matches the configured rate within tolerance, (b) no over-grant, (c) throughput is > 5M ops/sec on a modern laptop for tryAcquire.

Acceptance Criteria

  • Both algorithms implement the same RateLimiter interface.

  • Stress test with 100 concurrent threads shows no over-grant (total permits granted ≤ (rate × duration) + burst).

  • Under contention, tryAcquire throughput > 5M ops/sec (JMH-measured, see Phase 06 for JMH).

  • Rate limits are per-key and independent — hammering key A doesn’t slow down key B.

  • Idle keys are eventually evicted (implement TTL or Cleaner — unbounded map = OOM).

  • Exposes Micrometer MeterRegistry-compatible metrics (or write your own tiny metrics interface).

  • Written up: README explaining trade-offs between token bucket and sliding window, and why sliding window is more accurate but more memory-intensive.

What This Tests

Concurrent map usage, atomic primitives, CAS retry loops, per-key contention isolation, backpressure, and observability. This is essentially the code you’d write for an API gateway or an internal quota service. It also forces you to prove correctness with a stress test — you cannot claim rate limiting is correct without evidence.

Project B: Virtual Threads vs Thread Pool vs CompletableFuture Shootout

Time estimate: 12-16 hours

Problem Statement

Build a small HTTP client demo that fetches 10,000 URLs and measures wall-clock time, thread count, memory, and p99 latency under three concurrency models. Write it up like a Netflix Tech Blog post — numbers, graphs, and a recommendation.

Requirements

  • Target service: a local Spring Boot app with an endpoint that Thread.sleeps a random duration between 50-200 ms (simulating a slow downstream). Use MockServer, WireMock, or write it yourself.

  • Three client implementations, same input list of 10,000 URLs:

    1. Platform thread pool: Executors.newFixedThreadPool(N) where N is swept over {16, 64, 200, 1000}.

    2. Virtual threads: Executors.newVirtualThreadPerTaskExecutor().

    3. CompletableFuture: async pipeline on a fixed pool, non-blocking HttpClient (sendAsync).

  • Measure for each: wall-clock time, peak live thread count (jconsole or JMX), heap after run (jcmd GC.heap_info), and p99 request latency.

  • Introduce a synchronized block that does an I/O call somewhere in one variant. Show that under Java 21-23, virtual thread throughput collapses. Show that ReentrantLock fixes it.

Technical Constraints

  • Must use: java.net.http.HttpClient (JDK built-in), JFR to capture jdk.VirtualThreadPinned events for the virtual-thread run, -Djdk.tracePinnedThreads=short in the pinning demo.

  • Must NOT: call System.out.println inside the timed section (it’s synchronized — you’ll skew the numbers). Buffer output, print after.

  • Target JDK: 21+. If you can run 24+, run the same benchmark on both and quantify the pinning fix from JEP 491.

Acceptance Criteria

  • All three variants complete 10,000 requests successfully.

  • Report includes a table of wall-clock, peak thread count, and heap for each variant.

  • Report includes p99 latency — not just mean.

  • Report explicitly demonstrates the synchronized pinning problem on Java 21-23 with numbers.

  • Report includes a JFR flame graph or screenshot of the jdk.VirtualThreadPinned events (JMC or equivalent).

  • Report concludes with a decision table: when to use each.

  • Repo is public on GitHub with a README of your findings. This is a portfolio artifact.

What This Tests

Empirical thinking. study partners will ask “when do virtual threads help?” Anyone can quote the JEP. You’ll have measured it, and you’ll have the graphs. This is the project that answers the study question with numbers.

Project C: Deadlock Post-Mortem

Time estimate: 6-10 hours

Problem Statement

Given a deliberately deadlocked codebase (you’ll build the buggy version yourself, then debug it as if a colleague had written it), reproduce the deadlock, identify the root cause, propose two fixes, implement the better one, and write a post-mortem in the style of an incident review.

Setup

Build a small banking service with the classic ordering bug:

public class Account {
    private final long id;
    private BigDecimal balance;

    public synchronized void transfer(Account other, BigDecimal amount) {
        synchronized (other) {
            this.balance = this.balance.subtract(amount);
            other.balance = other.balance.add(amount);
        }
    }
}

Write a driver that spawns 50 threads, each performing random transfers between 10 accounts. Run it. It will deadlock, sometimes in 100ms, sometimes in 10s.

Requirements

  • Reproduce the deadlock reliably (either instrument with Thread.yield or run for at most 30 seconds).

  • Capture a jstack dump at the moment of deadlock and save it to the repo.

  • Annotate the dump: which two threads, which two monitor addresses, which two Account instances.

  • Propose two fixes:

    1. Global lock ordering by id.

    2. tryLock with timeout on a ReentrantLock, retry with random backoff.

  • Implement the ordering fix (it’s the more idiomatic one for this problem).

  • Write a jcstress test verifying no invariant violation under high concurrency (total balance across all accounts is invariant).

Deliverable: Post-Mortem

Write POST_MORTEM.md in the repo with these sections:

  1. What happened — symptoms, first alert, timeline

  2. Root cause — the lock ordering bug explained in ≤ 5 sentences

  3. Evidence — the annotated jstack dump

  4. Fix — what you changed and why

  5. Alternative fixes considered — the tryLock approach, its trade-offs

  6. Prevention — how to catch this in code review going forward (lock-ordering conventions, tools like -XX:+PrintConcurrentLocks, chaos tests)

  7. Verification — the jcstress test output showing no violations after the fix

Acceptance Criteria

  • The bug reproduces within 30 seconds on your machine, at least 5 out of 5 runs.

  • The jstack dump clearly shows the cycle.

  • The fix removes the deadlock (10,000 transfers run to completion in < 5 seconds).

  • Total balance is invariant across all transfers (prove it in the test).

  • Post-mortem reads like something you’d file at a real company — not a homework write-up.

What This Tests

Everything. Debugging under pressure, reading thread dumps, choosing between remediation strategies, and writing clearly about failure. This project is what you’d hand to an study partner as evidence you can survive a production incident.

Choosing Your Projects

Minimum: Complete A and C. Ship them. Recommended: Complete all three, in this order: A → C → B. A is a warm-up, C teaches you the tools, B applies them at scale.

If you finish all three, write about them. Not just READMEs — short blog posts. Publish them (or draft them and don’t publish; either way, writing forces clarity).

⚠️ What Most People Get Wrong

They write projects that “work” — the happy path completes. But study partners don’t care that your rate limiter grants a permit; they care whether it never over-grants under 10k concurrent producers. That’s the difference between “I built a rate limiter” and “I built a rate limiter I trust.” Every one of these projects has a stress-test acceptance criterion for exactly this reason. Do not skip them.

Return to README.md · Previous: 06_debugging_concurrency.md