Rung 7 (M11) — HARD GATE #2 — gRPC + REST ML Inference Service

HARD GATE

If this artifact is not shipped by end of Month 11, the roadmap PAUSES for 14 days, a post-mortem is written to ../99_pre_mortem/, and the artifact is rescoped and shipped before Month 12 begins.

Why this is a hard gate: this is the artifact that most directly maps to the 2026 applied-C++/ML job description. Job listings on Indeed and Built In SF as of mid-2026 explicitly ask for “experience building production inference services with vLLM, TGI, or NVIDIA Triton.” Rung 7 is your version of that experience. Without it, the capstones at M13 have no foundation, and your resume has no answer to the study question “tell me about a production system you’ve operated.”

Rung 7 is where you cross from “has C++ artifacts” to “has operated a C++ service.” A GitHub repo of code is a portfolio. A deployed URL you can hand an study partner is a product. This rung produces the latter.

What It Is

A public GitHub repo named mini-inference-cpp containing:

The Service

  • A C++20 inference server exposing both gRPC and REST endpoints.

  • Backend: ONNX Runtime for actual model execution. You are not writing your own inference engine here — you are writing the serving layer. This is a critical scoping decision.

  • Models served: at minimum, the MNIST MLP from Rung 6 exported to ONNX, plus one small pre-trained model (a distilled BERT or a small ResNet from the ONNX model zoo). Two models proves the multi-model story.

  • Dynamic batching: queue incoming requests up to N ms or B requests, whichever comes first, and dispatch as a batch to ONNX Runtime. Configurable via YAML or env vars.

  • Health / readiness endpoints: /healthz, /readyz on the REST side; standard gRPC health protocol on the gRPC side.

  • Prometheus metrics endpoint at /metrics, exporting: request count, request latency histogram (per-endpoint, per-model), inference latency (excluding queue time), queue depth, batch-size distribution.

  • OpenTelemetry tracing emitted to a configurable OTLP endpoint. Trace spans for: request received, batching queue wait, ONNX inference, response serialization.

  • Config-driven model loading (a simple YAML: model name, ONNX file path, input/output shapes).

The Delivery

  • Docker image on Docker Hub or ghcr.io, multi-stage build, final image < 300 MB.

  • Deployed to a $5/mo Linux VM: Hetzner CX11 (recommended, ~€4.51/mo) or DigitalOcean droplet ($6/mo). Reachable at a public URL.

  • Grafana dashboards (JSON committed to repo, screenshots in README):

    1. RED metrics dashboard: Rate, Errors, Duration.

    2. Per-endpoint latency dashboard: p50 / p95 / p99 for /predict at each model.

    3. Per-model throughput dashboard: requests per second, batch size histogram.

  • Load-test results from k6 or ghz at concurrency levels 1, 8, 32, 128, documenting p50 and p99 latency at each. Committed as a table in the README plus raw CSVs in benchmarks/.

The Communication

  • README with architecture diagram (blocks: client → load balancer → server → batcher → ONNX Runtime → model files).

  • Blog post on your personal blog covering: architecture decisions, why gRPC + REST, batching design, the actual latency numbers, what would break at 10× the load.

Why It Matters (Employer Signal)

One line: “Ships production inference infrastructure — the exact role I’m applying for.”

As of mid-2026, roles like ByteDance’s “Machine Learning Engineer - Inference,” F5’s “AI Inference Engineer,” GM’s “Senior ML Inference Engineer - Platform,” and every startup building an inference product all list the same competency stack: gRPC, ONNX Runtime or Triton, dynamic batching, Prometheus, OTel, Docker, cloud deploy. Rung 7 is a compressed evidence packet for that entire stack. No other rung on this ladder does this.

The deployed URL is the multiplier. Anyone can build a service on their laptop. A public URL an study partner can curl from their office proves you have crossed the operational threshold.

Acceptance Checklist — THE HARD-GATE CRITERIA

Functionality

  • Serves at least 2 ONNX models (the M9 MNIST MLP + one from the ONNX model zoo).

  • Both gRPC and REST endpoints work; documented in README with curl and grpcurl examples.

  • Dynamic batching implemented and configurable.

  • /healthz, /readyz, /metrics all functional.

  • Graceful shutdown: SIGTERM drains in-flight requests before exit.

Observability

  • Prometheus metrics endpoint scrapes clean.

  • Grafana dashboard JSONs committed under deploy/grafana/; three dashboards as listed above.

  • Screenshots of Grafana dashboards in README.

  • OpenTelemetry tracing verified against a Jaeger or Tempo backend (screenshots in README).

Performance & Load

  • Load test at concurrency 1 / 8 / 32 / 128 documented with p50 and p99 latency.

  • The results table is in the README.

  • Raw k6/ghz output CSVs committed in benchmarks/.

  • Honest reporting: if p99 spikes at concurrency 128, say so. Reviewers respect the truth.

Delivery

  • Multi-stage Dockerfile, final image < 300 MB.

  • Docker image published to ghcr.io or Docker Hub, tagged latest and by git SHA.

  • docker compose up in the repo brings up the server + Prometheus + Grafana on localhost.

  • Deployed to a public $5/mo VM. URL in the README. If cost is a concern, keep it running for at least 30 days after publication, then optionally take it down and replace the URL with a docker compose demo.

  • CI: build + test + Docker image build on every push.

Communication

  • README with architecture diagram.

  • Blog post published on personal blog, minimum 2000 words.

  • Blog post shared on r/cpp, r/devops, r/MachineLearning, and Hacker News (Show HN).

  • LinkedIn long-form post with the blog + repo + deployed URL.

  • License: MIT.

Where to Publish and Share

  • GitHub: public, MIT, topics: cpp, grpc, rest, onnx-runtime, inference-server, prometheus, opentelemetry, docker.

  • Docker Hub / ghcr.io: image published, README on Docker Hub explains basic usage.

  • Personal blog: the 2000-word writeup. This is the primary shareable link.

  • Hacker News: Show HN: I built a small production-shaped ML inference service in C++ to learn the stack. HN loves “built to learn.”

  • r/cpp — [Show r/cpp]. Lead with the architecture diagram.

  • r/devops — lead with the Grafana screenshots. Devops folks click for observability.

  • r/MachineLearning — flair [P]. Lead with the batching results.

  • LinkedIn: long-form. Frame as “12 months into my C++ mastery journey — here is the production artifact.” This is directly recruiter-facing.

  • Reply to job posts: when applying to inference roles, link this repo + the deployed URL in the cover letter. This is the point.

Reset Protocol — What Happens If This Ships Late

Same shape as Rung 6, tighter cuts.

  1. STOP. Do not begin Month 12 material.

  2. 14-day pause.

  3. Post-mortem in ../99_pre_mortem/. File 2027-XX-XX_m11_gate_miss.md.

  4. Rescope. The rescoped version drops, in this order:

    • Drop the second model (serve only the MNIST MLP).

    • Drop the OpenTelemetry tracing (Prometheus only).

    • Drop the $5/mo VM deploy (keep a docker compose demo).

    • Drop dynamic batching (single-request execution).

    • Keep: gRPC + REST, ONNX Runtime, Prometheus metrics, one Grafana dashboard, load test at 1/8/32, blog post, README with architecture diagram.

  5. Ship the rescoped version by day 14.

  6. Resume with M12 rescoped: the Rung 8 capstones absorb the missing bits (dynamic batching becomes a Capstone Alpha feature; the second model becomes part of Capstone Alpha).

If both hard gates miss (M9 and M11), the roadmap has failed as designed. This is diagnostic information about scope, time budget, or life circumstance — not a moral failure. Read ../99_pre_mortem/ in full and consider whether the 13-month plan needs to become an 18-month plan.

Common Ways This Rung Fails

  • You try to write your own inference engine. No. That is Capstone Beta territory. Here, ONNX Runtime is the backend. You are writing the serving layer.

  • Dynamic batching is a rabbit hole. Start with a naive time-window + max-batch implementation. Optimize only if you have slack. Do not read the Triton dynamic-batching paper before shipping v1.

  • You skip the deploy step. Then “production inference service” is a stretch on the resume. Deploy is what makes this rung honest. €4.51 a month is worth it.

  • You put Prometheus and Grafana in the docs but don’t run them. Then screenshots are missing and the observability signal collapses. Bring up the stack, take screenshots, commit them, then optionally tear the local stack down. The deployed VM should have the metrics endpoint scrapable at minimum.

  • You skip the load test. Then the README has no numbers, and the artifact is unverified. p50/p99 at concurrency 1/8/32/128 is the minimum credibility bar.

  • You use gRPC only, or REST only. Both endpoints. This is the small delta that shows understanding.

  • You forget graceful shutdown. SIGTERM handling that drops in-flight requests is a rookie signal. Handle it.

  • You over-engineer the config layer. YAML. Simple. Not a plugin system, not a service mesh integration. YAML.

What Most People Get Wrong

They build the service, run it on localhost, take screenshots, and call it shipped. But the whole point of Rung 7 is that it is operated, not just built. A publicly-reachable URL, running for 30 days, with metrics being scraped and dashboards updating in real time, is a categorically different artifact from a demo on your laptop. Pay the €4.51. That single decision determines whether Rung 7 is a portfolio piece or a hobby project.

Second common failure: they build all the observability and never look at it. Set up an alert on p99 latency or 5xx rate, even a laughable one (Discord webhook is fine), and mention it in the blog post. That single detail signals operational maturity.

Third: they write about the architecture but not about the incidents. If your service falls over during load testing at concurrency 128, that is content. If your first Docker image was 1.2 GB and you cut it to 280 MB, that is content. study partners eat this up.

Extension Challenges (Only After Base Ships)

  1. Model versioning: /predict/v1 and /predict/v2 with the same model at different weights. Nice for A/B story.

  2. Circuit breaker + timeout on the inference call. Real-world resilience.

  3. Kubernetes deploy: helm chart in deploy/helm/. Signal to devops-heavy shops.

  4. CUDA-enabled ONNX Runtime on a GPU VM (expensive; skip unless targeting GPU-inference roles).

  5. TensorRT backend as a plug-in alternative to ONNX Runtime. Nice differentiator; large scope.

  6. Autoscaling notes in the blog: what the batching curve looks like as concurrency grows, when you would horizontal-scale.

The first two (versioning, circuit breaker) are highest signal-to-effort.