01 — Capstone Alpha: MiniServe

MiniServe is a production-grade ML inference server written in modern C++. It reads ONNX models from disk, exposes gRPC + REST surfaces, does dynamic batching Triton-style, has full Prometheus + OpenTelemetry observability, and is load-tested against a Python FastAPI + PyTorch baseline. It is Capstone Alpha because it is the direct expansion of everything Phases 5 and 6 pointed at.

This is the capstone that says to a hiring team: I can build the thing you would otherwise pay Nvidia Triton or a full team six months to build a lite version of. It does not need to be as good as Triton. It needs to be shipped, benchmarked, and legible.

Positioning

Think of MiniServe as your professional-grade proof of work. The comparable products in the market are Nvidia Triton, TorchServe, KServe, BentoML, Ray Serve, and every ML-platform-team’s internal C++ inference server. You are not competing with them for market share. You are demonstrating that you understand what they do, that you can build a lite version, and that you can defend every design choice with numbers.

This is the part of the phase where you get to feel it: two years ago the idea of “you are shipping a Triton-lite” would have sounded absurd. Now it is your Saturday project. That is what all the prior phases were for.

Feature set

Must-have (defines success)

  1. ONNX model loading. Read .onnx files from a configured directory at startup. Multiple models loaded simultaneously.

  2. gRPC surface. Unary Predict, bidi PredictStream. Reflection enabled so grpcurl works out of the box.

  3. REST facade. drogon-based. /v1/models/{name}:predict (mimic TF-Serving path convention), JSON in, JSON out.

  4. Dynamic batching. Per-model config specifies max_batch_size and max_batch_delay_ms. Scheduler accumulates requests up to N or T ms whichever comes first, runs one batch through the model, fans out responses.

  5. Request queue with bounded size + backpressure. Overflow → RESOURCE_EXHAUSTED. Configurable per model.

  6. Prometheus metrics. RED per endpoint, per-model latency histograms, queue depth gauge, batch-size histogram, model-load time gauge.

  7. OpenTelemetry tracing. Each request produces a span; batch step is a child span with attribute batch_size.

  8. spdlog structured logs. JSON pattern, trace_id in every line.

  9. TLS everywhere. No cleartext ports on the deployed instance.

  10. Config file (YAML or JSON). Hot-reload on SIGHUP is a stretch; startup-only is the must-have.

  11. Docker image. Multi-stage build, final image < 250MB, published to a public registry.

  12. Deployment. Live on a small cloud VM. https://miniserve.<your-domain>.dev responds.

Non-features (explicitly NOT built)

Scope discipline is what makes this ship. Do NOT build:

  • Multi-node distributed inference. One VM. Sharding is a talking point, not a feature.

  • GPU support. ONNX Runtime’s CPU EP + AVX2/NEON is enough. Do not fight CUDA installation for a portfolio piece.

  • Model conversion or optimization (INT8, ONNX Runtime graph optimizations beyond defaults). Load, run, respond.

  • A web UI. README + grpcurl examples + one demo notebook is enough.

  • Custom model formats. ONNX only. If someone asks about TensorRT, you say “next iteration.”

  • User accounts / auth beyond a bearer token. Bearer token from env var. Nothing more.

  • Auto-scaling. One instance. kubectl is not your responsibility here.

  • Streaming LLM-style token generation. MiniServe is classification/regression/embedding-shaped. LLMs need KV cache and continuous batching — out of scope, and Beta might cover that ground if you pick that direction.

Write these non-features into the README explicitly. “What MiniServe is not” is one of the most credibility-boosting sections a portfolio project can have.

Architecture

[client] -- TLS gRPC --> [gRPC frontend (async CQ or callback API)] --.
                                                                     |
[client] -- HTTPS ------> [drogon REST facade] --> [gRPC internal] --'
                                                                     |
                                                                     v
                                                     [request queue (bounded, MPSC)]
                                                                     |
                                                                     v
                                                    [batching scheduler (per-model)]
                                                                     |
                                                                     v
                                                       [model runner pool (ONNX RT)]
                                                                     |
                                                                     v
                                                            [response fanout]

  Cross-cutting: spdlog (JSON) | prometheus /metrics :9091 | OTel OTLP-gRPC :4317

Draw this in excalidraw or mermaid. Save as docs/architecture.svg. Put it in the README’s second screenful.

Acceptance criteria

At W52, the following must all be true. This is the shipping bar for Alpha.

  1. Public URL responds. grpcurl -H "authorization: Bearer $TOKEN" <host>:443 miniserve.v1.Inference/Predict -d '{...}' returns a valid response.

  2. Multi-model. At least 2 models loaded (e.g., ResNet-50 + MobileNetV3). Both hit-testable.

  3. Benchmarks committed. ghz at concurrency 1, 8, 32, 128 for each model, both with and without dynamic batching. Six configurations, minimum. Numbers table in README.

  4. Python baseline benchmarked. Run the same models under FastAPI + PyTorch (or FastAPI + onnxruntime-python). Compare throughput and p99. You should win by >3x on throughput at concurrency 32+ or you have missed something — investigate before publishing.

  5. Grafana dashboard screenshot in README. RED + per-model throughput.

  6. Trace screenshot in README. One request from Jaeger showing gRPC span → batch span → model-inference span.

  7. Perf case study either included or referenced. P6.3 blog post link in the README.

  8. Deploy is one command. docker run or a justfile recipe. Non-obvious commands are documented.

  9. License = MIT or Apache 2.0.

  10. CONTRIBUTING.md exists, even if it just says “open an issue first.”

  11. Demo video / GIF under 60s. Screen recording of grpcurl request → Grafana dashboard tick → Jaeger trace opening. Loom or asciinema.

Benchmark methodology (the part you defend in studies)

Write the following into your benchmark README. study partners will ask about it, and having it pre-written signals seriousness.

  • Hardware. Cloud VM spec (vCPUs, generation, RAM). Note if it is shared vs dedicated.

  • Warmup. N seconds of load discarded before measurement.

  • Duration. 60s per configuration, minimum.

  • Repetitions. 3 runs per config; report median with min/max range.

  • Load generator. ghz for gRPC, k6 or oha for REST. Both committed.

  • Concurrency levels. 1, 8, 32, 128.

  • Metrics reported. RPS, p50, p95, p99, error rate.

  • Client and server co-located? State it. Ideally client on a separate VM in the same region.

  • Baseline. Python (FastAPI + PyTorch/ONNXRuntime) built with same care — do not sabotage the baseline. The credibility of your win depends on the baseline being reasonable.

A fair, well-documented benchmark that shows 3x is worth ten sloppy benchmarks showing 30x.

README structure (fill this exact skeleton)

# MiniServe

One-sentence pitch.

[architecture diagram]

[demo GIF]

## Why

2-3 paragraphs. Why does this exist, what problem does it solve, who is it for.

## Benchmarks

[headline number table: MiniServe vs Python baseline]

[per-concurrency table]

[link to benchmark methodology and raw data]

## Features

[bulleted list]

## Non-features (what MiniServe is NOT)

[bulleted list of anti-goals]

## Quick start

[docker run ... ]

[grpcurl example]

[curl example]

## Configuration

[YAML/JSON example with comments]

## Observability

[Grafana screenshot]

[Jaeger screenshot]

[how to plug into your own stack]

## Architecture deep-dive

[per-component walkthrough]

## Building from source

[cmake ...]

## Contributing

See CONTRIBUTING.md.

## License

MIT.

Blog post outline (published in M13)

Working title: “Building MiniServe: a 500-line-ish C++ ML inference server that beats FastAPI+PyTorch by 4x.”

Structure:

  1. The itch. Why you built it (“I wanted to understand what Triton does”).

  2. The design. The architecture diagram + why gRPC + why dynamic batching + why drogon for REST.

  3. The parts nobody talks about. Deadline propagation, bounded queues, backpressure. The distributed-systems basics from Phase 6.

  4. The observability wiring. How spdlog + prometheus + OTel actually connect. Screenshot of the working dashboard.

  5. The benchmarks. Table. Chart. Honest methodology paragraph.

  6. The surprises. The thing that took longest to debug (there will be one — tell that story honestly).

  7. What is next. The two things you would build first if you kept going. Signals humility and vision without being false-modest.

Aim: 2000-3000 words, 4-6 images, code snippets from the actual repo. Post on your own domain. Cross-post to r/cpp (Saturday show-off), HN (Show HN, weekday morning), and LinkedIn (weekday morning).

Failure modes (and how to save the project)

  • You can’t beat the Python baseline. Check: are you comparing async gRPC to sync FastAPI? Is dynamic batching turned on? Is the Python side using onnxruntime-python (fair) or PyTorch eager (unfair, but interesting). Adjust and re-run.

  • You blew W52. Cut features. Ship 2 models instead of 5. Ship without hot-reload. The absolute floor is: TLS gRPC + REST + Prom + one model + one benchmark = shippable.

  • Docker image is 2GB. Use a multi-stage build with debian:slim or distroless as the runtime base. Strip debug symbols. Static-link where possible.

What most people get wrong

  • They ship without a baseline comparison. Your 10K RPS number means nothing without the Python baseline’s 2.5K to compare against.

  • They over-optimize before shipping. Dynamic batching, allocator tuning, PGO — all go in AFTER v0.1 is live. Ship first, then optimize with data.

  • They hide the non-features. Say what you did NOT build. It shows scope discipline, which is a rare and prized trait.

  • They skip the architecture diagram. A hand-drawn or excalidraw diagram in the README doubles the perceived quality of the project in the reader’s mind. Draw it.