Phase 6 Projects — P6.1, P6.2, P6.3

Three projects. They stack: each one builds on the last. By W44 you have a running gRPC ML inference service with full observability and a published blog post explaining a real optimization you performed on it. That triple is the strongest resume artifact of Phase 6.

Each project has: acceptance criteria (binary pass/fail), stretch goals (nice-to-have), and anti-goals (things NOT to build; scope creep kills phases).

P6.1 — gRPC ML Inference Service (“miniserve”) — W40

One-line pitch: wrap the ONNX inference core from Phase 5.3 in a production-shaped gRPC service, TLS-terminated, deployed to a public $5/mo Linux VM, load-tested with real numbers in the README.

Scope

  • Load an ONNX model at startup (ResNet-50 or MobileNet as first target — keep it small).

  • Expose two RPCs: Predict (unary) and PredictStream (bidi streaming, used for dynamic batching).

  • Expose a REST facade on a separate port (drogon) for browser/curl clients. REST calls translate to internal gRPC.

  • Prometheus /metrics endpoint on port :9091.

  • Bounded request queue in front of the model. Overflow → RESOURCE_EXHAUSTED.

  • Deadline handling: server checks IsCancelled() between batch accumulation and inference.

  • TLS termination on the gRPC and REST ports (self-signed for dev, Let’s Encrypt on the VM).

  • Docker image published to Docker Hub or GHCR.

  • Deployed to a $5-10/mo Linux VM (Hetzner CX22, DO basic, or equivalent — avoid free-tier VMs, they steal CPU).

Acceptance criteria (all must pass)

  1. grpcurl against the deployed service returns a valid inference response over TLS.

  2. ghz load test at concurrencies 1, 8, 32, 128 for 60s each. Report p50/p95/p99 for each. Commit the raw JSON output to benchmarks/.

  3. /metrics returns a live histogram with at least one observation per label combination.

  4. Kill the VM’s network for 5s during a load test. Client sees UNAVAILABLE; service recovers within 2s of network return; no crash.

  5. Send a request with a 10ms deadline that requires 200ms of work. Server returns DEADLINE_EXCEEDED; queue does not leak.

  6. Send 10K requests in 1s to a queue sized 512. Excess returns RESOURCE_EXHAUSTED; server RSS does not grow beyond 2x the baseline.

  7. README includes: architecture diagram (a .svg — draw in excalidraw or mermaid), build instructions, run instructions, benchmark tables, sample curl and grpcurl invocations.

Stretch

  • Multi-model support: /models lists loaded models; each has its own metric labels.

  • Dynamic batching: accumulate up to N requests or T ms whichever comes first (Triton-style). Report throughput improvement vs no-batching.

  • Auto-detect the CPU (AVX2/AVX-512/NEON) and select the appropriate ONNX Runtime execution provider.

Anti-goals (do NOT do these in P6.1)

  • Do not implement your own model format. ONNX. Period.

  • Do not build a UI. This is a backend service.

  • Do not add authentication beyond a simple bearer token. OAuth is Phase 7 territory if at all.

  • Do not multi-node. One VM. Sharding is a Capstone Alpha concern.

Repo layout suggestion

miniserve/
  proto/miniserve.proto
  src/server_main.cpp
  src/service_impl.{h,cpp}
  src/rest_facade.{h,cpp}
  src/metrics.{h,cpp}
  src/model_runner.{h,cpp}
  benchmarks/ghz-*.json
  benchmarks/perf-*.data
  deploy/Dockerfile
  deploy/docker-compose.yml   # local stack: service + prom + grafana + jaeger
  README.md

Time budget

W37–W39 feed into this. W40 is the assembly and deployment week. Expect ~15–20 hours end-to-end. If it takes 30, ship it anyway — P6.2 depends on it.

P6.2 — Full Observability Harness — W42

One-line pitch: take P6.1, wire spdlog + prometheus-cpp + OpenTelemetry into it, spin up Grafana + Prometheus + Jaeger (or Tempo) as a Docker Compose stack, and build 3 dashboards that make it beautiful.

Scope

  • spdlog async, JSON pattern, rotating file sink + stdout, log level via env var.

  • Every request produces: one info log line at start, one at end (with latency), any warn/error inline.

  • prometheus-cpp: counters (requests, errors, retries), gauges (queue depth, models loaded), histograms (per-endpoint latency, per-model latency).

  • OpenTelemetry: traces on every RPC, span attributes for model name / batch size / status, OTLP-gRPC exporter to a local collector.

  • Docker Compose file bringing up: miniserve, prometheus, grafana, otel-collector, jaeger.

  • Grafana provisioned with 3 dashboards checked into git as JSON.

Acceptance criteria

  1. docker compose up produces a running stack. curl against miniserve returns a response; the request is visible as a Jaeger span and increments Prometheus counters.

  2. Dashboard 1: RED metrics. Rate, error %, p50/p95/p99 duration per endpoint. One row per endpoint. 30-min rolling window.

  3. Dashboard 2: per-endpoint latency. Heatmap of latency histogram, plus percentile lines overlaid.

  4. Dashboard 3: per-model throughput. Requests/sec by model label, queue depth gauge, model-load-time gauge.

  5. Take a screenshot of each dashboard under sustained load. Commit to docs/dashboards/.

  6. Trigger a fault (kill the model file, or set queue size to 1 and hammer it). Show the trace in Jaeger, the log line in stdout, and the error metric climbing — all three joinable by trace_id.

Stretch

  • Alerts: Prometheus alerting rules for p99 > 500ms for 5m and error_rate > 5% for 2m. Screenshot the firing state.

  • Loki for log aggregation. Cross-link Grafana log panel to trace panel via trace_id.

  • Metrics for allocator internals (jemalloc/mimalloc stats endpoint).

Anti-goals

  • Do not deploy the observability stack to prod. Local Docker Compose only. Grafana Cloud has a free tier; if you must expose dashboards, use that.

  • Do not build a custom exporter. OTLP is the standard; use it.

Time budget

~12–15 hours. Grafana dashboard tuning eats an unexpected amount — timebox each dashboard to 90 minutes.

P6.3 — Profiling Case Study Blog Post — W44

One-line pitch: pick one slow function in P6.1 or Phase 5, profile it end-to-end, optimize it, measure the delta, and publish the case study on your blog + r/cpp. This is the artifact that most changes how recruiters see you.

Scope

  • Choose the target function via profiling (perf, Tracy, or Instruments). Do NOT pick based on gut.

  • Establish a reproducible benchmark harness. Same input, same load, same hardware. Commit it.

  • Take the “before” measurement. p50/p99/mean/CV. Save flamegraph as before.svg.

  • Form a hypothesis about the fix. Write it down before implementing.

  • Implement one change. Not five.

  • Take the “after” measurement. Save after.svg.

  • Write it up. Structure:

    1. The service and the question (why does p99 matter here).

    2. The naive expectation (what you thought was slow).

    3. The profile (what was actually slow, with the flamegraph screenshot).

    4. The fix (the actual code change, with before/after snippets).

    5. The re-measurement (numbers table + after-flamegraph).

    6. What you learned (the delta between expectation and reality — the honest part).

Acceptance criteria

  1. Post is published on your own domain, not Medium. Use Hugo, Astro, or Zola — verify current SEO best practices in 08/03_public_signal_and_blog.md.

  2. Reproducible harness in a linked repo. A reader can git clone and reproduce your before/after within 20% variance.

  3. >2x speedup on the target metric (p99 latency OR throughput). If you get less, keep iterating; you have not found the real bottleneck yet.

  4. Post is submitted to r/cpp during a Saturday show-off thread. Also cross-post as a HN Show HN (Tuesday 8-10am ET is the golden window per HN analytics folk).

  5. LinkedIn post referencing the blog post, with the before/after flamegraph as the image.

Stretch

  • Second optimization on the same code, published as a follow-up.

  • Turn it into a 10-minute talk at a local C++ user group / meetup.

Anti-goals

  • Do NOT write a generic “here’s what perf is” tutorial. Every C++ blog has one. You are writing “here’s how I made my specific service 3x faster.”

  • Do NOT publish without real numbers. Fabricated or unrepresentative benchmarks will get you called out and remembered for the wrong reason.

  • Do NOT use synthetic microbenchmarks as the primary evidence. End-to-end p99 under representative load is the currency.

Time budget

~8–12 hours: 4 hours profile+fix, 4 hours writing, 2 hours polish, 1 hour cross-posting.

The Phase 6 exit portfolio

At W44, your GitHub should show:

  • miniserve — the service repo. README with benchmarks, architecture diagram, TLS deployment notes.

  • miniserve-observability — or a subdirectory — the Compose stack + dashboard JSON.

  • miniserve-perf-case-study — the blog-post repo (linked from your domain).

And on your blog: one post, real numbers, before/after flamegraphs.

That is what you take into Phase 7. Do not enter Phase 7 without it.