03 — ONNX Runtime C++

ONNX Runtime is the deployment default for PyTorch models in 2026. You train in Python with PyTorch, export to ONNX (torch.onnx.export), and run inference in C++ with ONNX Runtime — 50 MB of shared library instead of LibTorch’s 700 MB, ~5x faster startup, and a plug-in “execution provider” architecture that gives you CoreML on Mac, CUDA/TensorRT on NVIDIA, DirectML on Windows, and OpenVINO on Intel silicon, all behind the same C++ API. Microsoft maintains it, and the C++ API has been stable since 1.10 (currently 1.20+ in 2026). If you learn one C++ inference library for your career, this is it.

Your P5.3 project will be an ONNX Runtime HTTP inference server. This file gets you fluent enough to write it.

1. Install and CMake integration

Prebuilt binaries at https://github.com/microsoft/onnxruntime/releases. On macOS Apple Silicon:

# Download onnxruntime-osx-arm64-<version>.tgz, extract to /opt/onnxruntime/
curl -L https://github.com/microsoft/onnxruntime/releases/download/v1.20.1/onnxruntime-osx-arm64-1.20.1.tgz \
  | tar -xz -C /opt/
mv /opt/onnxruntime-osx-arm64-1.20.1 /opt/onnxruntime

vcpkg (recommended):

{
  "dependencies": ["onnxruntime"]
}

CMake (manual):

find_path(ORT_INCLUDE_DIR onnxruntime_cxx_api.h PATHS /opt/onnxruntime/include REQUIRED)
find_library(ORT_LIB onnxruntime PATHS /opt/onnxruntime/lib REQUIRED)

add_executable(hello_onnx src/hello_onnx.cpp)
target_include_directories(hello_onnx PRIVATE ${ORT_INCLUDE_DIR})
target_link_libraries(hello_onnx PRIVATE ${ORT_LIB})

CMake (via vcpkg):

find_package(onnxruntime CONFIG REQUIRED)
target_link_libraries(hello_onnx PRIVATE onnxruntime::onnxruntime)

2. The core types

ONNX Runtime’s C++ API is a thin C++ wrapper around a C API. The types you’ll use daily live in the Ort:: namespace:

Type	Purpose
Ort::Env	Global runtime state. One per process. Owns thread pools.
Ort::SessionOptions	Config: threading, graph optimization level, execution providers.
Ort::Session	A loaded model. Owns weights and execution plan.
Ort::MemoryInfo	Describes where a tensor lives (CPU, CUDA, arena).
Ort::Value	An input or output tensor (a variant).
Ort::AllocatorWithDefaultOptions	Allocator used by the API for output tensors.
Ort::RunOptions	Per-invocation config (log level, terminate).

3. Exporting a PyTorch model to ONNX (5 minutes in Python)

import torch
from torchvision.models import resnet18, ResNet18_Weights

model = resnet18(weights=ResNet18_Weights.DEFAULT).eval()
dummy = torch.randn(1, 3, 224, 224)

torch.onnx.export(
    model, dummy, "resnet18.onnx",
    input_names=["input"], output_names=["logits"],
    dynamic_axes={"input": {0: "batch"}, "logits": {0: "batch"}},
    opset_version=17,
)

opset_version matters — ONNX Runtime supports up to opset 21 in 2026; use ≥ 17 for modern models. dynamic_axes lets you accept variable batch sizes at inference time.

4. Minimal C++ inference program

#include <onnxruntime_cxx_api.h>
#include <array>
#include <iostream>
#include <vector>

int main() {
    // 1. Env: one per process.
    Ort::Env env(ORT_LOGGING_LEVEL_WARNING, "resnet18");

    // 2. Session options.
    Ort::SessionOptions opts;
    opts.SetIntraOpNumThreads(4);
    opts.SetGraphOptimizationLevel(GraphOptimizationLevel::ORT_ENABLE_ALL);

    // 3. Load the model.
    Ort::Session session(env, "resnet18.onnx", opts);

    // 4. Build the input tensor.
    std::array<int64_t, 4> input_shape{1, 3, 224, 224};
    std::vector<float> input_data(1 * 3 * 224 * 224, 0.5f);   // pretend-preprocessed image

    auto mem_info = Ort::MemoryInfo::CreateCpu(OrtArenaAllocator, OrtMemTypeDefault);
    Ort::Value input_tensor = Ort::Value::CreateTensor<float>(
        mem_info, input_data.data(), input_data.size(),
        input_shape.data(), input_shape.size());

    // 5. Names.
    const char* input_names[]  = {"input"};
    const char* output_names[] = {"logits"};

    // 6. Run.
    auto output_tensors = session.Run(
        Ort::RunOptions{nullptr},
        input_names,  &input_tensor,  1,
        output_names, 1);

    // 7. Read the output.
    float* logits = output_tensors[0].GetTensorMutableData<float>();
    auto shape = output_tensors[0].GetTensorTypeAndShapeInfo().GetShape();
    std::cout << "output shape: [";
    for (auto d : shape) std::cout << d << " ";
    std::cout << "]\n";
    std::cout << "logit[0..4] = " << logits[0] << " " << logits[1] << " "
              << logits[2] << " " << logits[3] << "\n";
}

That’s the whole inference story. Load, allocate input, run, read output.

5. Execution providers (EPs) — the reason to pick ONNX Runtime

An execution provider is a plugin that runs some or all of the graph on a specific backend. You add them in priority order; ORT partitions the graph and assigns each node to the first EP that can run it. Unassigned nodes fall back to CPU.

EP	Platform	Notes
CPUExecutionProvider	All	Always available. MLAS kernels (Microsoft’s BLAS-like).
CoreMLExecutionProvider	macOS 12+	Uses Apple’s Neural Engine / GPU. This is your Mac inference path.
CUDAExecutionProvider	Linux/Windows + NVIDIA	Uses cuDNN + cuBLAS.
TensorRTExecutionProvider	Linux/Windows + NVIDIA	JIT-compiles graph to TRT engine. Slower first run, faster steady state.
DirectMLExecutionProvider	Windows	DirectX-based GPU inference for AMD/Intel/NVIDIA.
OpenVINOExecutionProvider	Intel CPU/GPU/NPU	Intel-optimized.
QNNExecutionProvider	Qualcomm NPU	Snapdragon devices.
ROCMExecutionProvider	AMD GPUs on Linux	Analog of CUDA EP.

Adding an EP in C++

Ort::SessionOptions opts;
opts.SetGraphOptimizationLevel(GraphOptimizationLevel::ORT_ENABLE_ALL);

// CoreML on Mac
#ifdef __APPLE__
uint32_t coreml_flags = 0;   // 0 = default; see COREML_FLAG_* in coreml_provider_factory.h
Ort::ThrowOnError(OrtSessionOptionsAppendExecutionProvider_CoreML(opts, coreml_flags));
#endif

// CUDA on Linux
#ifdef USE_CUDA
OrtCUDAProviderOptions cuda_opts{};
cuda_opts.device_id = 0;
opts.AppendExecutionProvider_CUDA(cuda_opts);
#endif

Priority note: EPs are tried in the order you append them. Add the fast/specialized ones first (CoreML, then CUDA, then CPU as fallback), so ORT prefers them.

CoreML EP status on Apple Silicon (2026)

• Supports most vision and NLP transformer ops.

• Falls back to CPU for unsupported ops silently — ORT logs which nodes went where at ORT_LOGGING_LEVEL_INFO.

• Neural Engine only accessible via CoreML, not directly via C++. If your goal is “run this ResNet on ANE fast”, CoreML EP is the path.

• Speedup over CPU EP on Apple M2/M3: typically 3–8x for CNNs, 1.5–3x for transformers.

6. I/O binding — the perf trick

By default, when you call session.Run() with an input tensor on CPU and CUDA is your EP, ORT copies the input to GPU, runs, and copies the output back to CPU. On a hot inference path this copy is measurable.

I/O binding lets you preallocate output tensors on GPU and skip the transfers:

Ort::IoBinding binding(session);
binding.BindInput("input", input_tensor_on_gpu);
binding.BindOutput("logits", output_tensor_on_gpu);  // preallocated
session.Run(Ort::RunOptions{nullptr}, binding);

Not needed for CPU-only. Essential when you’re pushing throughput on CUDA/CoreML.

7. Threading options

Two knobs:

opts.SetIntraOpNumThreads(4);   // parallelism within a single op (matmul, conv)
opts.SetInterOpNumThreads(1);   // parallelism across independent ops in the graph

For a serving process handling many concurrent requests, use SetIntraOpNumThreads(1) and let your HTTP server handle concurrency. Otherwise every request contends for the same thread pool. This is the single biggest ORT-configuration mistake in production.

8. Batching

Two strategies:

Static batching: the client sends inputs of shape [batch, ...]. Your server forwards it. Simple, wastes latency if the client sends 1 at a time.

Dynamic batching: your server has a small queue. It waits up to max_wait_ms (typically 5–20 ms) collecting inbound requests, then packs them into a single [batch, ...] tensor, runs one session.Run(), and demuxes the outputs back to the callers.

Sketch for P5.3:

struct Request { std::vector<float> input; std::promise<std::vector<float>> reply; };
std::queue<Request> queue;
std::mutex m; std::condition_variable cv;

void batcher_thread(Ort::Session& session) {
    while (true) {
        std::vector<Request> batch;
        {
            std::unique_lock<std::mutex> lk(m);
            cv.wait_for(lk, std::chrono::milliseconds(10),
                        [&]{ return !queue.empty(); });
            while (!queue.empty() && batch.size() < 32) {
                batch.push_back(std::move(queue.front()));
                queue.pop();
            }
        }
        if (batch.empty()) continue;

        // Pack batch → single tensor → run → demux → set promises.
    }
}

Dynamic batching gives dramatic throughput wins under concurrent load (2–5x is normal), at a cost of adding p50 latency by the wait time. Report both metrics.

9. Comparison: ONNX Runtime vs alternatives (2026)

Runtime	Strength	Weakness
ONNX Runtime	Broad EP support, stable C++ API, good docs, MSFT-backed	Ops sometimes lag frontier PyTorch by a release
TensorRT (NVIDIA)	Fastest steady-state inference on NVIDIA hardware	NVIDIA-only, verbose C++ API, engine build is slow
OpenVINO (Intel)	Best perf on Intel CPU/GPU/NPU	Intel-focused; ORT’s OpenVINO EP is usually enough
MLC LLM / TVM	Compiler-first, cross-hardware, LLM-focused	New, less docs, requires compilation step
llama.cpp (LLMs)	Small, quantized, everywhere	LLM-only
CoreML (direct)	Direct Apple Silicon ANE access	Apple-only, Objective-C++ API
TFLite	Mobile/embedded default	TensorFlow ecosystem

Rule of thumb: ORT is the default. Reach for TensorRT only if you’ve measured the 20–40% additional NVIDIA speedup and you’re NVIDIA-locked. Reach for MLC LLM or llama.cpp if you’re serving LLMs on-device.

10. Loading a real model — one worked example

// ---- Load ResNet-18 exported from PyTorch ----
Ort::Env env(ORT_LOGGING_LEVEL_WARNING, "rn18");
Ort::SessionOptions opts;
opts.SetIntraOpNumThreads(1);
opts.SetGraphOptimizationLevel(GraphOptimizationLevel::ORT_ENABLE_ALL);

#ifdef __APPLE__
Ort::ThrowOnError(OrtSessionOptionsAppendExecutionProvider_CoreML(opts, 0));
#endif

Ort::Session session(env, "resnet18.onnx", opts);

// Query I/O programmatically instead of hardcoding names:
Ort::AllocatorWithDefaultOptions alloc;
auto in_name  = session.GetInputNameAllocated(0, alloc);
auto out_name = session.GetOutputNameAllocated(0, alloc);

Programmatic name queries are important because ONNX export names can vary and hardcoding “input”/”logits” breaks when a colleague re-exports.

11. What most people get wrong

• They leave IntraOpNumThreads at the default (equal to CPU cores) and then wonder why concurrency 32 doesn’t scale — every request contends for all cores. Set to 1 for servers.

• They don’t set ORT_ENABLE_ALL optimization level. Default is basic. Missing 20–40% of achievable throughput.

• They forget to warmup the session. First session.Run() compiles kernels and populates arenas — 5–10x slower than steady state. Do 5 warmup runs before measuring.

• They benchmark with session.Run() on their laptop and don’t include the CoreML EP. Then they conclude ORT is slow. Add CoreML on Mac, CUDA on Linux — that’s the fair comparison.

• They append EPs in the wrong order (CPU before CoreML). CPU claims all ops. CoreML never runs.

• They hardcode input/output names instead of GetInputNameAllocated. Breaks when the model is re-exported.

• They use session.Run with fresh input allocation on every call. Reuse buffers.

• They ignore the fact that ONNX opset lags PyTorch. A model using scaled_dot_product_attention in PyTorch 2.11 might export cleanly at opset 21 but be sub-optimal at opset 17 — check.

12. Practice exercises

1. Export a ResNet-18 to ONNX with dynamic batch axis. Load and run in C++ with a batch of 4 real preprocessed images. Verify the top-1 predictions match the Python version.

2. Add CoreML EP on your Mac. Measure inference time before and after. Report the ratio.

3. Enable ORT_LOGGING_LEVEL_INFO and read the log — find where ORT logs which nodes went to which EP.

4. Build the dynamic batcher sketch in section 8 as a standalone. Feed it 100 concurrent requests. Measure throughput at batch sizes 1, 8, 32.

5. Export a small BERT (or DistilBERT) and run it in C++. Measure CoreML vs CPU EP latency.

---

Nav: ← 02 LibTorch · Next: 04 pybind11 and nanobind →