09 · The ML Serving C Stack¶
When a request hits a production LLM in 2026, it travels through five to seven layers, most of which are C or C++ under a Python or Go shell. This file surveys the four systems every applied-ML engineer should know exists — NVIDIA Triton Inference Server, ONNX Runtime, TensorRT / TensorRT-LLM, and llama.cpp — and tells you which parts of each are C, which are C++, and where you’d actually plug in.
The four stacks, side by side¶
System |
Owner |
Primary language |
Front-end |
Where the C is |
License |
|---|---|---|---|---|---|
Triton Inference Server |
NVIDIA |
C++ |
gRPC / HTTP / Python client |
Core scheduler, backend API (C) |
BSD-3 |
ONNX Runtime |
Microsoft (open gov’d) |
C++ |
Python, C, C#, Java, JS |
Public C API ( |
MIT |
TensorRT / TensorRT-LLM |
NVIDIA |
C++ |
Python builder API, C++ runtime |
Runtime plugins, in-flight batching |
Proprietary (some OSS wrappers) |
llama.cpp / llama-server |
ggml-org |
C / C++ |
HTTP (OpenAI-compat) |
Everything, top to bottom |
MIT |
These are not competitors so much as a stack: Triton orchestrates, ONNX Runtime or TensorRT-LLM or llama.cpp is a backend inside Triton, and each of those backends contains its own hand-written CUDA and CPU kernels.
Triton Inference Server — the orchestrator¶
Triton (github.com/triton-inference-server/server) is NVIDIA’s model-serving daemon. It fronts multiple model backends (TensorRT, ONNX Runtime, PyTorch, Python, vLLM, TensorRT-LLM, ensemble) with a unified HTTP/gRPC API. Its value proposition: dynamic batching, model concurrency, GPU scheduling, model repository management, all in one binary.
The piece you’d read as a C/C++ practitioner:
src/core/— the dispatcher, request queue, batching logic.The Backend API (
triton_backend.h) — a C interface, deliberately, so any language that can produce a shared library can implement a backend. This is the ABI seam.
You will not casually contribute to Triton in your first year of systems C. But you will deploy it at Zoho for a Zia service if the workload justifies GPU serving. Know it exists; know it is the reference orchestrator.
Note: do not confuse NVIDIA Triton Inference Server with OpenAI Triton — a Python-embedded DSL for writing GPU kernels. Same word, different projects. The Triton in this file is the server.
ONNX Runtime — the portable runtime¶
ONNX Runtime (github.com/microsoft/onnxruntime) is the closest thing to a lingua franca for ML inference. Export a model from PyTorch to ONNX, then run it anywhere ORT runs — CPU (with MLAS kernels), CUDA, TensorRT, DirectML on Windows, CoreML on Apple, OpenVINO on Intel, WebNN in browsers.
The C API is stable and documented: onnxruntime_c_api.h. Every language binding sits on top of this. Skim it once — the pattern (create environment, create session, allocate tensors, run) is what every inference C API looks like:
OrtEnv *env;
g_ort->CreateEnv(ORT_LOGGING_LEVEL_WARNING, "myapp", &env);
OrtSession *session;
g_ort->CreateSession(env, "model.onnx", options, &session);
// build input OrtValue, run, read output OrtValue
MLAS (Microsoft Linear Algebra Subprograms) is ORT’s CPU kernel library — hand-tuned SIMD for GEMM, activations, quantized ops. It lives at onnxruntime/core/mlas/ and is worth grepping for mlas_gemm_kernel. Same design as OpenBLAS/BLIS from 04_matmul_and_gemm.md, specialized for NN shapes.
TensorRT / TensorRT-LLM — the NVIDIA lock-in¶
TensorRT is NVIDIA’s optimizing compiler + runtime for GPU inference. You give it an ONNX or a native definition; it fuses ops, picks the best kernels per GPU, calibrates INT8/FP8/FP4, and hands back an engine file specialized for one specific card. TensorRT-LLM (github.com/NVIDIA/TensorRT-LLM) is the LLM-focused wrapper: paged KV cache, in-flight (continuous) batching, LoRA hot-swap, speculative decoding, quantization down to FP4 on Blackwell.
The C++ runtime API is the piece a systems engineer touches; the Python builder API is what MLEs use to compile. TensorRT-LLM’s cpp/include/tensorrt_llm/runtime contains the scheduling loop, KV cache manager, batch manager — that is where the interesting C++ lives.
On Blackwell (2025+), TensorRT-LLM ships FP4 kernels that on Llama-3-70B roughly double throughput vs FP8 at comparable quality. The pattern is now: bigger model, more aggressive quant, memory-bandwidth pays for it — same argument as ggml, different hardware.
llama.cpp / llama-server — the pragmatic full stack¶
See 06_reading_llama_cpp.md for the codebase tour. The relevant point here: llama-server is a single-binary, drop-in OpenAI-compatible endpoint. Continuous batching, quantized KV cache, embedded model, no external dependencies. It has quietly become the default “run this LLM behind an API” tool for anything that fits on a single node.
How it sits in the stack:
Alone, on a laptop or single server → replaces the entire stack for small/mid deployments.
Behind Triton as a custom backend → gives you Triton’s batching + llama.cpp’s kernels. Rarer.
Wrapped by Ollama or LM Studio → the same binary, prettier UX.
The C ABI seam is the whole point¶
Why is every one of these frameworks’ public API in C, not C++? ABI stability. C has a stable ABI on every platform — you can compile Triton with GCC 11, load an ORT backend built with Clang 17, load a TensorRT plugin built with the NVIDIA compiler, all in one process. C++ ABI is compiler- and stdlib-version-locked; the moment you #include <string> in a public header, you have committed to a specific libstdc++ or libc++ ABI. This is why every serious framework’s public surface is extern "C" structs and function pointers, even when the internals are C++. Understanding this constraint is what separates a library user from a library author. See Phase 7 · 04_portability_and_abi.md.
Comparative throughput (rough, mid-2026, single-node)¶
Setup |
Model |
Hardware |
Tokens/sec |
Source |
|---|---|---|---|---|
llama.cpp Q4_K_M |
Qwen3.5-35B |
RTX 3090 |
~100 (tuned) |
community bench |
llama.cpp defaults |
Qwen3.5-35B |
RTX 3090 |
~50 |
“ |
Ollama (llama.cpp) |
Qwen3.5-35B |
RTX 3090 |
~15–20 |
“ |
vLLM (PyTorch + custom kernels) |
Llama-3-70B FP16 |
4× A100 80G |
~3000 aggregate |
vLLM bench |
TensorRT-LLM FP4 |
Llama-3-70B |
H100 |
~2500 single-GPU |
NVIDIA blog |
Triton + TRT-LLM |
Llama-3-70B FP8 |
8× H100 |
10k+ aggregate |
NVIDIA blog |
Benchmarks age fast — the point is orders of magnitude, not exact numbers. What holds: hand-tuned kernels + right precision + right batching >> generic runtime + FP16.
Where you plug in¶
Contributing to llama.cpp — achievable in M11 with the reading from 06_reading_llama_cpp.md. Model architecture addition (see Discussion #16770), quantization backend, or a small perf PR.
Contributing to ONNX Runtime — harder but tractable; the codebase is enormous but well-documented. Good first issues appear regularly.
Contributing to Triton (NVIDIA) — gated by needing NVIDIA hardware and deep familiarity with their internal patterns. Low ROI for a self-directed learner.
Contributing to TensorRT — not open-source. Skip.
What most people get wrong about the ML serving stack¶
They think there’s one “correct” serving framework and the others are legacy. There isn’t. Each solves a different constraint: Triton is a datacenter orchestrator, ONNX Runtime is a portability layer, TensorRT is an NVIDIA-specific speed weapon, llama.cpp is the everything-in-one-binary answer for small deployments. Real production stacks use two or three of them together — Triton in front, ORT or TRT-LLM as the backend, llama.cpp for the internal experiments and CPU fallback. Learn the shape of each, know which one to reach for, and remember that C is the ABI seam that lets them coexist in one process at all.
Return to README.md · Next: projects.md