Phase 5 — C++ for ML and Data (Months 7-9, Weeks 25-36)

This is the phase that answers “why did you spend a year on C++?” Every hour of Phases 1-4 was preparation for the twelve weeks in front of you. You are here to become the C++ engineer that ML teams reach for when Python is not enough — when a training loop’s data path has become the bottleneck, when a model needs to be served at sub-10ms p50 latency, when a research prototype has to be wrapped as a pip install-able package for the rest of the org, when a new custom operator needs to slot into PyTorch’s dispatcher. That is your target job. That is what this phase makes you credible for.

You have Python-ML instincts. You know what a training loop, a dataloader, a tensor, a batch, an epoch, a p99 latency graph feels like. You have never wired those instincts to C++. That’s what Phase 5 fixes. By W36 you will have four public projects that make the story clean: a from-scratch neural net in Eigen matching PyTorch to within 1%, a pybind11 wrap of it published to PyPI, an ONNX Runtime C++ inference server benchmarked against Python FastAPI, and an Arrow data pipeline that beats pandas by 5x. Those four repos, together, are the study.

Mission

Turn “someone who knows C++” into “someone applied ML teams hire”. You are learning the six tools that show up on every ML-platform C++ job description:

  1. Eigen for numerical linear algebra when you can’t (or don’t want to) pull in LibTorch.

  2. LibTorch (PyTorch C++) for custom ops, model surgery, and rare in-C++ training.

  3. ONNX Runtime C++ for the actual production inference path in 2026.

  4. pybind11 / nanobind for Python ↔ C++ bindings — the single most common bridge you’ll build.

  5. Apache Arrow C++ for zero-copy columnar data pipelines that beat pandas.

  6. CUDA basics so you can read and lightly modify GPU kernels; MPS / cloud GPU workarounds because you’re on Apple Silicon.

Plus one architectural theme: model serving. Triton internals, custom serving, latency vs throughput trade-offs, batching strategies.

Exit criteria — you’re done with Phase 5 when

  1. You can build a two-hidden-layer MLP in Eigen from scratch (forward + backprop + SGD) and train it on MNIST to > 95% test accuracy.

  2. Your Eigen MLP matches a PyTorch reference to within 1% on the same seed and hyperparameters. You have a written explanation for any residual gap.

  3. You can wrap a C++ class as a Python module with pybind11 or nanobind, expose numpy-interop, correctly release the GIL for long computations, and publish to test-PyPI.

  4. You understand the pybind11 vs nanobind trade-off in one sentence and can defend the choice for a given codebase (existing PyTorch-adjacent → pybind11; new project on Python 3.12+ → nanobind).

  5. You can load a PyTorch-exported ONNX model in C++ using ONNX Runtime and run inference with correct output on at least one CV or NLP model.

  6. You can select an ONNX Runtime execution provider by platform (CPU / CoreML on Mac / CUDA on Linux) and explain latency differences.

  7. You have written or read a TORCH_LIBRARY custom op registration and know what the PyTorch dispatcher does.

  8. You can read an Apache Arrow schema, build a RecordBatch from raw pointers with zero copy, and execute a filter/project pipeline via arrow::compute.

  9. You know how to read a Parquet file into Arrow with memory-mapping and how that saves RAM vs pd.read_parquet.

  10. You have written a CUDA kernel (even a “vector-add”) on a cloud GPU (RunPod or Vast.ai) and can explain __global__, thread blocks, warps, and shared memory at a first-principles level.

  11. You have working MPS-backed PyTorch on your Mac, and you know its limits (op coverage gaps).

  12. You have built an ONNX Runtime C++ HTTP inference server (via drogon or httplib) with batching, and benchmarked its p50/p99 latency and throughput at concurrency 1/8/32 against a Python FastAPI baseline.

  13. You can articulate when C++ serving wins over Python (small models, tight tail-latency budgets, many concurrent connections) and when it does not (large LLMs where the bottleneck is the GPU, not the host language).

  14. You have published all four Phase 5 projects to public GitHub repos with README, benchmarks, and (for at least one) a blog post.

  15. You have submitted a job application referencing at least one of these projects. This is the point of the phase.

Weekly breakdown — W25 to W36

Week

Focus

Deliverable

W25

Eigen: matrices, vectors, expression templates, aliasing, Map<> for raw pointers

Notes + toy problems (dot product, matrix multiply, solve Ax=b)

W26

Overlap with Phase 4 P4.2 finalization; start Eigen MLP forward pass

Forward pass matches PyTorch on a hand-checked example

W27

Backprop derivation on paper, then in Eigen

Backprop matches PyTorch autograd on same seed

W28

Full training loop, MNIST > 95%

P5.1 v1.0 shipped

W29

pybind11 deep dive: numpy interop, GIL, buffer protocol, exceptions

Toy mypkg binding a sum_vec() C++ function

W30

nanobind comparison + scikit-build-core packaging

Understand the trade-off first-hand

W31

Wrap P5.1 as a pip-installable package

P5.2 shipped, on test-PyPI

W32

LibTorch: tensors, modules, TorchScript loading; ONNX Runtime C++ setup

hello_torch.cpp and hello_onnx.cpp compile-run-return-a-tensor

W33

ONNX Runtime server v0: single-request inference

HTTP POST /infer returns correct predictions

W34

Add batching, benchmarking, Prometheus metrics

p50/p99 measured at concurrency 1/8/32

W35

P5.3 shipped — README, graphs, blog post

Post to r/MachineLearning + LinkedIn

W36

Arrow C++: RecordBatch, Parquet, Compute; P5.4 shipped

Beat pandas by ≥ 5x on 1M-row filter/project

Slack: 1-2 buffer days per week. You will need them.

Employer signal — which companies, which roles

The four projects hit these targets directly:

  • NVIDIA (Triton / CUDA / DL frameworks teams) — LibTorch, ONNX Runtime, custom ops, CUDA read-fluency.

  • Meta (PyTorch core, ExecuTorch, MLPerf teams) — pybind11, TORCH_LIBRARY, dispatcher understanding.

  • Google (TensorFlow, JAX, MediaPipe) — Bazel-adjacent, Eigen (they use it internally), pybind11.

  • Microsoft (ONNX Runtime, DirectML) — direct hit on ONNX C++ knowledge.

  • Apple (MLX, CoreML, ML Compute) — nanobind (MLX uses it), Metal / MPS, Apple Silicon perf.

  • Databricks / Snowflake / Dremio — Arrow, Parquet, compute engines, Bazel.

  • Anthropic / OpenAI / xAI infra teams — C++ serving, low-latency inference, pybind11.

  • Hugging Face (candle team, transformers infra) — inference C++, ONNX, tokenizer bindings.

  • Indian: Ola Krutrim, Sarvam AI, Zoho MLplatform, Fractal, Mu Sigma — ML platform engineers who need to move heavy pipes off Python.

study types you’re now credible for:

  • Systems + ML combined: write a memory-efficient dataloader in C++ that streams Parquet into a training loop.

  • Custom op: write a PyTorch op in CUDA / C++ and expose it to Python.

  • Latency debugging: given a serving stack with unexpected tail latency, hypothesize and profile.

  • Take-home: “wrap this NumPy function in C++ for a 10x speedup” — Eigen + pybind11 is the answer.

What most people get wrong

They learn LibTorch API syntax without understanding the dispatcher and ATen. They write torch::matmul(a, b) in C++, get output, and stop. Then in an study when asked “when you register a TORCH_LIBRARY op, at what level of the stack does dispatch happen? What’s the difference between CPU and CUDA kernel selection?” they freeze.

The dispatcher is the routing layer that maps at::add to the right kernel based on tensor device, dtype, layout, and autograd state. ATen is the tensor library beneath it. TORCH_LIBRARY is how you plug in a new op that participates in that routing. Understanding these three primitives is the difference between “someone who wrote a LibTorch demo” and “someone who could actually work on PyTorch”. Do that reading (pytorch/aten/src/ATen/native/README.md is the entry point) — 90 minutes, one Saturday, huge return.

Second thing they get wrong: they treat pybind11 vs nanobind as a taste question and never think about it. It is not. It is a Python-ABI question. nanobind targets the Stable ABI on Python 3.12+, so the same wheel works across Python versions. pybind11 does not (wheels are Python-version specific). If you are shipping a package, this is a big deal. Know it.

Third: they build a “C++ inference server” that outperforms Python by 3x on a laptop with a batch of one — and then don’t test it under actual concurrent load, where its advantage may be much larger (or, on GPU-bound big models, negligible because the GPU is the bottleneck). Always benchmark at concurrency 1, 8, and 32. Report all three.

Fourth: they use pandas for data prep in a C++ project (“just do it in Python first”), then wonder why the overall pipeline is slow. Arrow C++ + Parquet is what the modern data stack looks like. Use it early.

How to work this phase

  • Two anchor sessions per week (2-3 hours each): one for reading + notes, one for project work.

  • Weekend deep-work (4-6 hours): the four projects live here.

  • Follow the roadmap week-by-week; don’t skip forward. The Eigen MLP is the mental model foundation for everything after — if you can’t do it, the Torch dispatcher and Arrow layout won’t stick.

  • Every project on the P4.1 template. Zero infrastructure work in Phase 5. All 10-15 hours/week go to ML content.

  • Blog as you go. One post per project. Even a 500-word “what I learned” moves you from “someone who did tutorials” to “someone who writes about their work” — hugely different signals to a hiring manager.

You’re not learning C++ any more. You’re learning to be a C++ ML engineer. Different skill, different bar. Meet it.

Nav: ← Phase 4 Projects · Next: 01 Eigen →