Case Studies & War Stories — Real Systems, Real Scars

Reading production post-mortems is the fastest way to build the intuition that separates a mid-level engineer from a senior one. Every case below is public — blogged, keynoted, or open-sourced by the teams involved. This file distills five to eight of them with the same pattern: what they built, why, what went right, what bit them, and the study-ready takeaway.

Use these as mental reference points. When someone asks you a design question, the honest answer is often “Team X at $BigCo tried exactly this and here’s what happened.”

1. Netflix — Hystrix, Then Its Retirement

Context: By 2011 Netflix was running hundreds of services in AWS. Cascading failures between them were routine. Ben Christensen’s team built Hystrix — a Java library implementing the circuit breaker pattern with a thread-pool bulkhead and a real-time dashboard.

What went right:

  • Made the circuit breaker a mainstream Java pattern. Every subsequent Java resilience library owes a debt.

  • The dashboard was the first time most Java devs saw their downstream call topology.

  • Forced explicit thinking about failure modes. “What if the recommendations service is slow?” got a first-class answer.

What bit them:

  • Thread-pool-per-dependency was expensive; hundreds of pools per service.

  • The API was heavyweight and intrusive — every downstream call wrapped in a HystrixCommand subclass.

  • By 2018 Netflix put Hystrix in maintenance mode and moved to adaptive concurrency limits (their concurrency-limits library) plus Resilience4j for standalone use.

study takeaway: the circuit breaker pattern is now the industry default (Resilience4j today), but the bulkhead has moved from thread pools to semaphores and adaptive limiters. When someone asks “have you used Hystrix?” — the right answer is “Hystrix is deprecated; the current standard is Resilience4j, and Netflix themselves have moved on. Same pattern, better implementation.”

2. LinkedIn — Kafka’s Origin Story

Context: By 2010 LinkedIn was drowning in point-to-point integrations. Every new service meant N new pipes. Jay Kreps, Neha Narkhede, and Jun Rao built Kafka to unify the event backbone. It was open-sourced in 2011, went to Apache in 2012, and became the de facto event backbone of the industry.

What went right:

  • Log-structured, replayable, high-throughput. All three at once, which was novel.

  • Simple mental model: an append-only distributed log. Users can reason about it.

  • Consumer groups: multiple systems can independently read the same stream.

What bit them (and everyone else):

  • Operating Kafka at scale is genuinely hard. ZooKeeper (until KRaft in 2.8+) was a source of persistent pain.

  • Exactly-once semantics arrived late (0.11, 2017) and are subtle to configure correctly.

  • Rebalancing storms during consumer restarts — cooperative rebalancing (2.4+) helps but doesn’t eliminate.

study takeaway: Kafka won because it was the honest answer to a real integration problem, not because it was the fastest. When you argue for Kafka in a design study, argue for the log abstraction and replayability — those are the differentiators over RabbitMQ / SQS / NATS.

3. Twitter — The Java Migration Reality

Context: Twitter’s original monolith was Ruby on Rails. By 2011–2013, the “fail whale” era, they migrated the core timeline path to JVM (Scala mostly, some Java) with Finagle as their RPC library and Mesos for scheduling.

What went right:

  • Latency dropped by an order of magnitude at scale that Rails couldn’t reach.

  • The JVM ecosystem (async I/O via Netty, thread management, GC tuning) let them run at hyperscale without a rewrite every 18 months.

  • Open-sourced Finagle, Twemproxy, and later Twemcache — the Twitter stack shaped the industry.

What bit them:

  • Scala’s compile times were brutal and pushed some teams back toward Java.

  • The finagle-heavy architecture made simple things ceremonious. New engineers took months to become productive.

  • The custom RPC / discovery / config stack accumulated its own gravity. Migrations off Finagle to gRPC only started years later.

study takeaway: JVM was the right long-term bet for a company at Twitter scale, but the “custom stack” cost is real. Prefer boring standards (Spring, gRPC, Kafka) unless you have Twitter’s scale problems.

4. ING Bank — Spring Boot at Regulated Scale

Context: ING has been public about running thousands of Spring Boot microservices across their banking platform, with a mix of reactive (WebFlux) and traditional MVC. Their engineering blog and conference talks (SpringOne, Devoxx) document the reality.

What went right:

  • Spring Boot’s opinionated defaults let hundreds of teams ship consistently — same actuator endpoints, same metrics format, same security posture.

  • Reactive stack (WebFlux + Reactor) for gateway / aggregation layers, MVC for CRUD-heavy back-office. Not “reactive everywhere.”

  • Heavy use of Spring Cloud Config, Spring Cloud Gateway, Resilience4j — the boring, well-supported pieces.

What bit them:

  • Reactive debugging is genuinely harder. Stack traces are unreadable, Mono.flatMap chains hide errors, backpressure semantics are subtle. Teams that adopted reactive “because it scales” without a scaling problem regretted it.

  • Migrating from Spring Boot 1.x → 2.x → 3.x is a real project each time. Property renames, deprecated auto-configurations, Jakarta EE 9 namespace change.

  • Spring Cloud Config was a single point of failure early on.

study takeaway: reactive Java is a specialist tool, not a default. Say this out loud: “I’d use WebFlux for a gateway or an aggregator with high fan-out I/O; I’d use MVC (with virtual threads on Java 21+) for everything else. Reactive debugging is genuinely harder and the payoff needs to be real.”

5. Uber Michelangelo — ML Platform at Scale

Context: Uber’s Michelangelo (2015+) is one of the earliest published ML platforms. Its serving layer combines a Scala/Java stack for online prediction with Python for training. Some components are open (Horovod, Ludwig); the core platform is proprietary but well-documented in the Uber engineering blog.

What went right:

  • Clear separation between training (Python, Spark, TensorFlow) and serving (JVM, low-latency).

  • Feature store as a first-class primitive — the same features used at training time are available at inference time, versioned.

  • Model registry, versioning, and rollback baked into the platform.

What bit them:

  • Feature/serving skew — training features and serving features must be identical. Bugs where they diverge are common and painful.

  • Model refresh cadence vs online serving stability — a stale model is often better than a shakily-deployed new one.

  • Explaining model decisions (regulatory / customer support) required a whole subsystem.

study takeaway: the feature store is the single most important idea from Michelangelo. When you talk about serving ML in production, mention feature/serving skew as the top risk. That signals you’ve done more than a tutorial.

6. LinkedIn Photon-ML — Classical ML in the JVM

Context: LinkedIn’s Photon-ML (open source, ~2016) is a Spark-based library for generalized linear models with random effects. It served the “who should I recommend to whom” problems at LinkedIn’s scale. All JVM (Scala on Spark).

What went right:

  • Kept the entire pipeline — training and inference — in one runtime. No PythonJVM handoff.

  • Random-effect models let them personalize per user without training a separate model per user.

  • Open-sourced with a real Apache-2 license, not “source available.”

What bit them:

  • Being JVM-only meant they couldn’t easily use PyTorch / TensorFlow innovations as the field moved deep. Later systems added Python interop.

  • Scala/Spark ML is niche today; a new hire needs training on the stack.

  • Feature engineering DSLs in Scala are powerful but foreign to most ML engineers coming from pandas/NumPy.

study takeaway: JVM-native ML works for classical models (GLM, GBDT, random forests). Once you need modern deep learning, you’re better off exporting to ONNX or serving from Triton. Photon-ML is a good case study when someone argues “you can’t do serious ML in Java” — you can, for the ML that most business problems actually need.

7. Monzo — Microservices at 2000+ Services

Context: UK challenger bank Monzo has been remarkably public about their 2000+ microservice architecture (Go and Java mix, mostly Go). Their post-mortems on major outages are required reading.

What went right:

  • Aggressive service isolation kept blast radius small. When one service failed, the bank as a whole degraded but didn’t stop.

  • Standard toolkit (same libraries, same observability, same deploy pipeline) across all services. New services shipped in hours.

What bit them:

  • The 2019 outage was famously caused by a Cassandra config change that cascaded — because every service ultimately depended on the same Cassandra cluster. “Isolated” services with a shared dependency aren’t isolated.

  • The cognitive overhead of 2000+ services is real. Even with great tooling, understanding the whole system is beyond any one person.

study takeaway: microservices give you deploy independence and failure isolation only if the underlying dependencies are also isolated. A shared database that all services talk to is a distributed monolith wearing a costume. Reference the 2019 Monzo outage when discussing this — it’s the canonical case study.

8. Airbnb — Java Migration From Rails (and the Reactive Trap)

Context: Airbnb’s core platform was Rails through the mid-2010s. Search, payments, and other performance-critical paths migrated to a JVM stack (mostly Java with some Kotlin, using Dropwizard and later Spring).

What went right:

  • 10× throughput improvements on the paths that were migrated.

  • Better observability out of the box (JVM ecosystem: JFR, async-profiler, Micrometer).

  • Rails services still exist for admin / non-critical paths — pragmatic mixed stack.

What bit them (and every large migration):

  • Migration takes years, not months. Teams work in both stacks simultaneously.

  • Feature parity is a moving target — the Rails app keeps evolving while you migrate.

  • The reactive experiment (some teams tried RxJava heavily) was quietly rolled back for the same reason ING found: debug cost > perf benefit outside gateways.

study takeaway: big-bang rewrites fail; strangler-fig migrations succeed. When asked about legacy modernization, say “I’d wrap the legacy service in an API gateway, migrate one endpoint at a time, and dual-run with metrics comparison. Full rewrites are how you lose two years and your best engineers.”

The Cross-Cutting Patterns

Reading these together, the same lessons keep appearing:

  1. Failure isolation is the whole point of distribution. Shared dependencies (DB, cache, ZK, config service) will always be the actual limit.

  2. The right abstraction beats the fast abstraction. Kafka won on log semantics, not TPS. Spring won on defaults, not raw speed.

  3. The JVM is a serious ML serving runtime. It is not a serious ML training runtime. Design the boundary.

  4. Reactive is a scalpel, not a hammer. Use it where fan-out I/O dominates. Don’t use it because a conference talk was persuasive.

  5. Boring wins over 5 years. Spring Boot, Kafka, Postgres, Redis, Kubernetes. Every “we invented our own X” case study has a “we regretted inventing our own X” chapter.

Practice — Read One Post-Mortem a Week

Rotate through these sources. When something catches your eye, spend 30 minutes writing your own 500-word summary. Doing this weekly for a year gives you a corpus you can draw on in any senior study.

The engineers who get promoted are the ones who can say “I remember the Monzo Cassandra outage — we should watch for that here.” That memory only exists if you read the stories now.

Return to README.md · Previous: 06_cloud_native_kubernetes.md · Next: projects.md