Projects · Phase 5 Portfolio Anchors¶

These four projects are why you did the last 24 weeks of work. They are what recruiters at NVIDIA, Meta, Databricks, HuggingFace, Anthropic, Ola-Krutrim, Sarvam, and Zoho ML Platform actually want to see. Each is small enough to finish in 2–4 weekends and specific enough to prove a real skill. Publish every one to a public GitHub repo with a README, benchmarks, and (for at least two) a short blog post. study partners will find them.

The portfolio arc¶

Each project maps to a Phase 5 file and a specific study signal.

#	Project	File	Weeks	Signal to employer
P5.1	Eigen tiny NN from scratch	01	W28–W30	“You understand backprop and modern C++ numerics without a framework.”
P5.2	pybind11 wrap of P5.1	04	W31	“You can ship a C++ core as a pip-installable Python library.”
P5.3	ONNX Runtime inference server	03, 07	W33–W35	“You have written the kind of service they pay you to write.”
P5.4	Arrow data pipeline	05	W36	“You understand modern columnar data systems, not just models.”

Do them in order. Each builds on the last. By W36 you have four public repos that read together as one story: numerics → bindings → serving → data.

P5.1 · Eigen tiny NN from scratch (W28–W30)¶

Goal¶

Build a two-layer MLP in modern C++ with Eigen. Train it on MNIST. Match a PyTorch reference to within 1 % test accuracy on the same random seed. Benchmark it.

Minimum viable spec¶

Two dense layers, ReLU hidden, softmax output.
Forward pass and analytical backward pass, written by hand.
SGD with momentum, or Adam. Pick one and get it right.
Loads MNIST from IDX files (no framework helpers).
20 epochs, batch size 128, > 95 % test accuracy.
Compiled -O3 -march=native with Apple Accelerate as the BLAS backend.
Benchmark using Google Benchmark: forward-pass time per sample.

Acceptance criteria¶

README with: how to build, how to run, what accuracy to expect, and a plot of loss over epochs.
A python/reference.py script that trains the same architecture in PyTorch with the same seed and prints its final test accuracy. Your C++ version must be within 1 % of that number.
Google Benchmark output committed as bench.txt. Should show something like < 50 µs per forward pass for batch 1, < 500 µs for batch 128 on M-series.
CI (GitHub Actions matrix from Phase 4 template): builds on Ubuntu clang-18 and macOS clang, all sanitizers clean.

Traps to avoid¶

Row-major vs column-major. Eigen defaults to column-major; PyTorch is row-major. If your weight matrix layout does not match your reference, your accuracy will silently drift by 3–5 %. Use Eigen::Matrix<float, Dynamic, Dynamic, RowMajor> throughout.
Backprop off-by-one. Cross-entropy + softmax has a clean combined gradient; do not compute them separately. Write a unit test that finite-differences your gradient on random 3-parameter inputs.
Numerical instability. Compute log-softmax with the max-subtraction trick or you will NaN on the second epoch.
Timing under -O0. Every bench must be -O3. Turn that on in the Google Benchmark preset.

GPU-optional variant¶

On a rented RTX 4090 (see file 06): port the forward pass to a small CUDA kernel and register it via TORCH_LIBRARY. Compare against torch.nn.Linear at batch 512. This becomes a talking point in studies — “I ported my Eigen backend to CUDA and it was 8× faster at batch 512.”

Publish¶

Public GitHub repo: raghul/tiny-eigen-nn (or similar).
README with badges, build instructions, benchmark table, and PyTorch parity plot.
One tweet-length LinkedIn post or blog: “Backprop by hand in 500 lines of Eigen — what I learned.”

Employer signal¶

Hiring managers looking for “applied C++ for ML” scan for exactly this: a small, honest, well-tested numerical kernel with real benchmarks. It answers the question “can this person do C++ math?” more clearly than any resume line.

P5.2 · pybind11 wrap of P5.1 (W31)¶

Goal¶

Expose P5.1 as a Python package. pip install tinynn-cpp gives you a working import tinynn; m = tinynn.MLP(784, 128, 10); m.fit(X, y). NumPy arrays go in and out zero-copy.

Minimum viable spec¶

pyproject.toml using scikit-build-core.
pybind11 module exposing MLP, fit, predict, save, load.
NumPy inputs mapped with py::array_t<float, py::array::c_style | py::array::forcecast>.
py::gil_scoped_release around the training loop.
Wheels built via cibuildwheel for macOS-arm64 and manylinux-x86_64.
Published to TestPyPI (not real PyPI — the point is the process, not squatting on a name).

Acceptance criteria¶

pip install --index-url https://test.pypi.org/simple/ tinynn-cpp works on both macOS and Linux.
A Jupyter notebook in the repo (demo.ipynb) that installs the package, trains on MNIST, plots the confusion matrix.
pytest suite covers: (a) forward pass matches C++ output within 1e-6, (b) fit reduces loss, (c) save/load round-trips.
Zero-copy verified: run np.shares_memory(X, tinynn.internal_last_input()) and get True after training.
Optional: nanobind branch on a separate git branch, showing the same module in nanobind with the compile-time / binary-size delta measured in the README.

Traps to avoid¶

Building wheels on GitHub Actions. cibuildwheel is the answer. Do not hand-roll wheel builds in bash.
Forgetting the GIL release. Without it, calling fit from Python blocks other threads for the entire training run. Users notice.
Copying arrays on input. If forcecast triggers a copy (wrong dtype from Python side), you lose the zero-copy claim. Assert dtype in the C++ side or document clearly.
__reduce__ / pickling. If you skip it, users cannot ship models across processes with multiprocessing. Add minimal pickle support.
ABI drift. If P5.1 changes its class layout, your wheel breaks silently. Pin the git submodule SHA.

GPU-optional variant¶

Add a device="cuda" argument that dispatches to your CUDA kernel from P5.1’s optional variant. Requires only that you feature-flag it in CMake.

Publish¶

Same repo as P5.1, add python/ subdirectory and TestPyPI badge.
Blog post: “Shipping a C++ NN as a pip-installable wheel: scikit-build-core, pybind11, cibuildwheel end-to-end.” This kind of post ranks well on Google and gets cited by other people building bindings. Free credibility.
Optional stretch: publish the nanobind branch as tinynn-cpp-nb on TestPyPI and compare.

Employer signal¶

Everyone with an ML infra role posts “Python-first with C++ hot paths” in their JD. Nobody proves they can do it. A pip-installable wheel with a working demo notebook proves it in five minutes of a recruiter’s time.

P5.3 · ONNX Runtime inference server (W33–W35)¶

Goal¶

Build a small HTTP inference server in C++ that loads a PyTorch-exported ONNX model, serves it, and benchmarks head-to-head against a Python FastAPI + PyTorch baseline.

Minimum viable spec¶

Model: ResNet-18 image classifier, or a small transformer (< 50 M params). Export from PyTorch to ONNX with opset 17+ and dynamic batch axis.
Server: drogon (or cpp-httplib if you want simpler). Endpoints: POST /predict (JSON), POST /predict_image (multipart PNG/JPEG), GET /metrics (Prometheus).
Preprocessing in C++: resize, normalize (mean/std), NCHW layout. Use stb_image for image decode.
Dynamic batching per section 3c of file 07: max_batch=32, max_delay=5 ms.
Prometheus metrics: requests_total, latency_seconds histogram, batch_size histogram, inflight_requests gauge.
Dockerized. Multi-stage build. Final image < 250 MB.
Benchmarks with wrk2 at concurrency 1, 8, 32, and 128. Same benchmark script runs against your service and against a Python FastAPI + PyTorch reference.

Acceptance criteria¶

README with a benchmark table:
- Rows: your service (no batch), your service (batched), Python baseline.
- Columns: RPS, p50, p95, p99, memory footprint (RSS).
Latency histogram plot (matplotlib) generated from wrk2’s HDR output.
Grafana dashboard JSON committed under dashboards/. Screenshot in the README.
CI runs the C++ build, ONNX model download from an S3 URL or a GitHub Release asset, and a smoke test (one request).
Docker image published to GitHub Container Registry.

Traps to avoid¶

Forgetting SetIntraOpNumThreads(1). File 07 section 9. This alone can 5× your throughput at high concurrency.
Timing warmup. First 100 requests always slower. Discard them in the benchmark.
Load generator on the same box. Run wrk2 from a separate container or machine.
Bench-optimizing for RPS while ignoring p99. Report both, and pick a headline number that reflects real production (throughput at p99 < 50 ms is a common way).
Not saving the ONNX file in the repo. Push it via Git LFS or a Release asset. Model download instructions should be one line.
Comparing against slow Python. Your baseline should be a reasonable FastAPI + uvicorn --workers 4 + torch.jit.traced model. Do not benchmark against Flask single-worker — that is not honest.

GPU-optional variant¶

Rent an RTX 4090 or H100 for a weekend. Rebuild your image with onnxruntime-gpu and enable the CUDA execution provider. Re-run all benchmarks. Publish a second table. Talking point: “CUDA EP + FP16 gave me 12× throughput at the same p99.”

Publish¶

Public GitHub repo: raghul/ort-cpp-serving-lab (or similar).
Docker image on GHCR.
Blog post is mandatory for this one. Title suggestion: “How much faster is C++ inference than Python FastAPI, actually? A benchmark story.” Include the graphs. This is the piece recruiters will link to internally.

Employer signal¶

This is the flagship. It says: I can take a PyTorch model, ship it as a C++ service with proper metrics, and measure it honestly. That is the job. Every ML infra manager reading this repo learns everything they need to know in ten minutes.

P5.4 · Arrow data pipeline (W36)¶

Goal¶

Build a C++ ETL that reads Parquet, filters and transforms via Arrow Compute, writes Parquet. Benchmark against a Pandas equivalent. Beat Pandas by 5× on 1 M rows.

Minimum viable spec¶

Input: a Parquet file with 1 M rows, 10 columns (mix of int64, float, string, timestamp).
Pipeline:
1. Read Parquet with column projection (only load the 5 columns you need).
2. Filter rows on a numeric threshold.
3. Compute a new column via Arrow Compute (multiply, then cast to float32).
4. Group by a string column and aggregate (sum, mean).
5. Write output to a new Parquet with ZSTD compression.
Streaming variant: process in RecordBatch chunks of 10 k rows so peak RSS stays under 100 MB regardless of input size.
Pandas reference (reference.py) does exactly the same steps.
Benchmark harness times both, reports wall time and peak RSS.

Acceptance criteria¶

Correctness: outputs from C++ and Pandas match byte-for-byte after canonicalization (sort by key, cast to same dtypes).
Throughput: C++ is at least 5× faster than Pandas on 1 M rows for the whole pipeline.
RSS: streaming variant stays under 100 MB on a 100 M-row input; Pandas OOMs at that scale, which is the point.
README with input schema, pipeline diagram (ASCII is fine), and benchmark table.
Bonus: expose the pipeline via pybind11 as mypipe.run(input_path, output_path) that returns a pyarrow.Table of the aggregated result zero-copy.

Traps to avoid¶

ValueOrDie() in prod. Use ARROW_ASSIGN_OR_RAISE in Status-returning functions. Aborting on a bad Parquet file is not acceptable.
Reading all columns. Column projection is the entire point. Verify with strace -e read that you are not touching bytes for unread columns.
Materializing full Table when you should stream. For 100 M rows the streaming variant is mandatory.
Comparing Snappy against no-compression. Snappy is usually faster end-to-end. Compare Snappy to ZSTD-3 for realistic tradeoffs.
Pandas benchmark using object dtype for strings. That is a strawman. Use pd.ArrowDtype for the string column so the comparison is fair.
Skipping Acero. For the group-by-aggregate step, Acero is the right tool. Hand-rolled aggregation is a red flag in code review.

GPU-optional variant¶

Swap Arrow Compute for cuDF (RAPIDS) on a rented GPU. Same pipeline. Expect 10–30× over pandas, but honestly report GPU memory-transfer overhead — it dominates for small inputs.

Publish¶

Public GitHub repo: raghul/arrow-cpp-pipeline.
README with the benchmark table and a plot of RSS vs input size (C++ streaming stays flat, Pandas curves up and OOMs).
Optional blog post: “Arrow C++ vs Pandas on 100M rows: when serialization is your bottleneck.”

Employer signal¶

Most “ML engineer” candidates cannot articulate why data pipelines exist. Someone who has built an Arrow C++ ETL and can talk about columnar layout, streaming, zero-copy, and Acero stands out at Databricks, Snowflake, Ray, Dremio, Spice.ai, and every data-heavy AI product team.

Cross-cutting rules for every project¶

One uniform standard for the four repos. This is what turns four demos into a portfolio.

Reuse the Phase 4 template. All four repos are forks of your P4.1 CMake/Conan/gtest/benchmark/CI template. study partners who see the same layout across all four know you have a real system, not four one-offs.
Every repo has a README with: what it is (one paragraph), how to build (three commands), how to run (one command), a benchmark table with numbers on a named machine (“M3 Pro, 18 GB, macOS 15.3”), a link to the Phase 5 file it maps to.
All benchmarks in a bench/ directory with a Makefile target make bench that regenerates them.
CI green on the badge. A red badge is worse than no badge.
License MIT or Apache-2.0. Recruiters need to know they can point their engineers at your code.
Cite what you reused. Eigen, ONNX Runtime, drogon, Arrow are open source. Credit them in the README.
Cap README at 3 screens. More belongs in a blog post or the code itself.

Suggested weekly cadence¶

Weekend	Focus
W28	P5.1 skeleton: repo, CMake, Eigen types, MNIST loader, forward pass
W29	P5.1 backprop, training loop, PyTorch reference parity
W30	P5.1 benchmarks, README, publish, LinkedIn post
W31	P5.2 pybind11 wrap, scikit-build-core, cibuildwheel, TestPyPI
W32	Buffer week: revise P5.1/P5.2 based on any peer feedback, one blog post
W33	P5.3 skeleton: ORT + drogon + Docker + one endpoint
W34	P5.3 batching, Prometheus, Grafana, wrk2 harness
W35	P5.3 Python baseline, benchmarks, blog post
W36	P5.4 Arrow pipeline, streaming, Pandas comparison, publish

That is a defensible C++-for-ML portfolio built in nine weekends. It is enough to open doors at every company in this file’s employer table.

Nav: ← 07 model serving in C++ · Phase 6 (M10+, TBD) →