04 — pybind11 and nanobind

This is the single most important file in Phase 5 for your career. “Applied ML engineer who can drop into a C++ backend and expose it to Python cleanly” is a hire-signal that scales across every ML infra team — PyTorch, JAX, TensorFlow internals, NumPy plugins, custom PyTorch ops, ONNX Runtime plugins, Ray workers, TensorRT plugins, TF Lite delegates. pybind11 is the incumbent (used by PyTorch’s own Python bindings). nanobind, by the same author (Wenzel Jakob), is the 2022 rewrite — ~4x faster compile, ~5x smaller binaries, ~10x lower runtime overhead — and it’s what MLX (Apple), JAX, PennyLane, LLVM/MLIR, and Dr.Jit have migrated to. In 2026, new projects should use nanobind if they can target Python 3.12+; existing pybind11 code stays pybind11 because migration is real work. Learn both.

At the end of this file you will be able to (a) wrap a C++ class as a Python module, (b) accept and return NumPy arrays with zero-copy, (c) correctly release the GIL for long-running C++ work, (d) package the module with scikit-build-core and publish to PyPI, and (e) explain the ABI/perf trade-off between pybind11 and nanobind in one paragraph.

1. Install

# pybind11 (via pip):
pip install pybind11

# nanobind (via pip):
pip install nanobind

# Or via vcpkg:
#   "dependencies": ["pybind11", "nanobind"]

CMake:

# pybind11
find_package(pybind11 CONFIG REQUIRED)
pybind11_add_module(_mypkg src/bindings.cpp)

# nanobind
find_package(nanobind CONFIG REQUIRED)
nanobind_add_module(_mypkg src/bindings.cpp)

Both macros handle Python discovery, module suffix (.cpython-312-darwin.so), and link flags for you.

2. Hello, pybind11

// bindings.cpp
#include <pybind11/pybind11.h>
namespace py = pybind11;

int add(int a, int b) { return a + b; }

PYBIND11_MODULE(_mypkg, m) {
    m.doc() = "mypkg: my first C++ ↔ Python bridge";
    m.def("add", &add, "add two ints", py::arg("a"), py::arg("b"));
}

Build it into _mypkg.cpython-312-darwin.so, drop it on sys.path, and:

import _mypkg
_mypkg.add(2, 3)   # 5
help(_mypkg.add)   # shows the docstring

3. Exposing a class

#include <pybind11/pybind11.h>
#include <pybind11/stl.h>       // std::vector <-> list, etc.
namespace py = pybind11;

class Counter {
public:
    explicit Counter(int start = 0) : n_(start) {}
    int  incr()          { return ++n_; }
    int  value() const   { return n_; }
    void reset(int v)    { n_ = v; }
private:
    int n_;
};

PYBIND11_MODULE(_mypkg, m) {
    py::class_<Counter>(m, "Counter")
        .def(py::init<int>(), py::arg("start") = 0)
        .def("incr",  &Counter::incr)
        .def("reset", &Counter::reset, py::arg("v"))
        .def_property_readonly("value", &Counter::value)
        .def("__repr__", [](const Counter& c) {
            return "<Counter n=" + std::to_string(c.value()) + ">";
        });
}

Usage:

from _mypkg import Counter
c = Counter(10)
c.incr(); c.incr(); print(c.value)   # 12

4. NumPy interop — the reason you’re really here

The payoff of C++ bindings is bulk numerical work. py::array_t<float> accepts a NumPy array, gives you a raw pointer, and (with the right flags) does it zero-copy.

#include <pybind11/pybind11.h>
#include <pybind11/numpy.h>
namespace py = pybind11;

// Sum an array in C++. Zero-copy input; scalar return.
double sum_array(py::array_t<float, py::array::c_style | py::array::forcecast> arr) {
    auto buf = arr.request();     // ask for a raw buffer view
    float* data = static_cast<float*>(buf.ptr);
    size_t n = buf.size;

    double acc = 0.0;
    for (size_t i = 0; i < n; ++i) acc += data[i];
    return acc;
}

// Return a new NumPy array from C++.
py::array_t<float> squared(py::array_t<float, py::array::c_style | py::array::forcecast> arr) {
    auto buf = arr.request();
    float* in = static_cast<float*>(buf.ptr);

    // Allocate a NumPy-owned output.
    py::array_t<float> result(buf.shape);
    float* out = static_cast<float*>(result.request().ptr);

    for (py::ssize_t i = 0; i < buf.size; ++i) out[i] = in[i] * in[i];
    return result;
}

PYBIND11_MODULE(_mypkg, m) {
    m.def("sum_array", &sum_array);
    m.def("squared",   &squared);
}

Usage:

import numpy as np, _mypkg
x = np.arange(1_000_000, dtype=np.float32)
_mypkg.sum_array(x)        # 499999500000.0
_mypkg.squared(x[:5])      # array([0., 1., 4., 9., 16.], dtype=float32)

Flags matter:

  • py::array::c_style — require C-order (row-major); if the input is Fortran-order, pybind11 will complain.

  • py::array::forcecast — if the input is float64, cast it to float32 (allocates a temporary copy). Omit if you want a hard error on wrong dtype.

Bridging to Eigen:

#include <pybind11/eigen.h>   // one include, and Eigen::MatrixXf is auto-converted from/to np.ndarray

Eigen::VectorXf normalize(const Eigen::VectorXf& x) {
    return x / x.norm();
}

PYBIND11_MODULE(_mypkg, m) {
    m.def("normalize", &normalize);
}

Zero-copy conversion happens when the NumPy array is C-contiguous with matching dtype. This is the shortest bridge between NumPy and Eigen you will ever write.

5. The GIL — the bug that ruins your first release

The Global Interpreter Lock is held by Python whenever CPython bytecode runs. When your C++ function is called, the GIL is held by default. If your function takes 500 ms, no other Python thread makes progress for 500 ms. Users on ThreadPoolExecutor see zero parallelism.

Release the GIL for long-running C++ work:

double long_computation(py::array_t<float> arr) {
    auto buf = arr.request();
    float* data = static_cast<float*>(buf.ptr);
    size_t n = buf.size;

    double acc = 0.0;
    {
        py::gil_scoped_release release;    // drop the GIL for the crunchy part
        for (size_t i = 0; i < n; ++i) acc += std::sqrt(data[i]);
    }
    // GIL is reacquired here — needed if you touch py::* objects.
    return acc;
}

Rules:

  1. Release the GIL only when you are done touching Python objects.

  2. Do not touch any py:: object while the GIL is released — it will segfault.

  3. If you need to call Python from a released region, reacquire with py::gil_scoped_acquire.

Free-threading (Python 3.13+ no-GIL) note (2026): free-threaded CPython eliminates the GIL entirely. pybind11 supports it in 2.13+; nanobind supports it natively. In free-threaded builds, gil_scoped_release becomes a no-op. Your code doesn’t break — the release is just unnecessary. Continue writing GIL-release blocks for portability.

6. nanobind — the modern choice

The surface API is deliberately similar to pybind11 to ease migration. Same file rewritten in nanobind:

#include <nanobind/nanobind.h>
#include <nanobind/stl/vector.h>
#include <nanobind/ndarray.h>
namespace nb = nanobind;

int add(int a, int b) { return a + b; }

nb::ndarray<float, nb::shape<-1>, nb::c_contig> squared(
        nb::ndarray<float, nb::shape<-1>, nb::c_contig> arr) {
    size_t n = arr.shape(0);
    float* out = new float[n];
    for (size_t i = 0; i < n; ++i) out[i] = arr(i) * arr(i);

    // Ownership: return the array with a capsule owner so it's freed when Python is done.
    nb::capsule owner(out, [](void* p) noexcept { delete[] static_cast<float*>(p); });
    return nb::ndarray<float, nb::shape<-1>, nb::c_contig>(out, {n}, owner);
}

NB_MODULE(_mypkg, m) {
    m.def("add",     &add);
    m.def("squared", &squared);
}

Differences you’ll actually feel:

Concern

pybind11

nanobind

Compile time (one file, ~500 LoC)

~10 s

~2.5 s

.so size for that module

~800 KB

~150 KB

Function-call overhead

~1 µs

~100 ns

Python ABI target

Version-specific

Stable ABI (Py 3.12+) — one wheel works for 3.12, 3.13, 3.14

ndarray interop

NumPy only via py::array_t

Framework-agnostic (NumPy / PyTorch / JAX / TF)

Free-threading (no-GIL)

Supported since 2.13

Native, more efficient

Ecosystem maturity

Everywhere (PyTorch, SciPy, many libs)

MLX, JAX, LLVM/MLIR, PennyLane, Dr.Jit — growing fast

When to pick which

  • Existing pybind11 codebase: stay. Migration is real work with limited return unless build-time or wheel-size is a specific problem.

  • Existing PyTorch-adjacent code (writing a custom PyTorch op via torch/extension.h): stay with pybind11 — PyTorch’s own bindings are pybind11 and interop is easier.

  • New project, Python 3.12+: nanobind. Faster builds, smaller wheels, cleaner ndarray. Especially if you’re shipping wheels to users who span multiple Python versions.

  • New project, must support Python 3.11 or older: pybind11 (nanobind requires 3.9+, and stable-ABI wheels want 3.12+).

The ndarray framework-agnostic interop

This is nanobind’s signature feature. Same C++ function accepts NumPy, PyTorch, JAX, or TF tensors via DLPack — the industry standard for zero-copy tensor sharing.

// Accept any framework's tensor of shape (N, D), float32, on CPU.
void process(nb::ndarray<float, nb::shape<-1, -1>, nb::device::cpu> t) {
    for (size_t i = 0; i < t.shape(0); ++i)
        for (size_t j = 0; j < t.shape(1); ++j)
            t(i, j) *= 2.0f;   // in-place, zero copy
}

import numpy as np, torch, jax.numpy as jnp
_mypkg.process(np.zeros((3, 4), dtype=np.float32))    # works
_mypkg.process(torch.zeros(3, 4))                     # works
_mypkg.process(jnp.zeros((3, 4)))                     # works

One binding, four framework users. This is what MLX and JAX are exploiting.

7. Packaging with scikit-build-core

Modern Python packaging for C++ extensions. Killed setup.py. Uses pyproject.toml + CMake.

Project layout

mypkg/
├── CMakeLists.txt
├── pyproject.toml
├── README.md
├── src/
│   ├── bindings.cpp
│   └── core/           # your "real" C++ code
└── python/mypkg/
    ├── __init__.py     # re-exports from _mypkg
    └── py.typed        # PEP 561 marker for type hints

pyproject.toml

[build-system]
requires = ["scikit-build-core>=0.9", "nanobind>=2.0"]
build-backend = "scikit_build_core.build"

[project]
name = "mypkg"
version = "0.1.0"
requires-python = ">=3.12"
dependencies = ["numpy"]

[tool.scikit-build]
minimum-version = "0.9"
cmake.version = ">=3.28"
wheel.packages = ["python/mypkg"]

CMakeLists.txt

cmake_minimum_required(VERSION 3.28)
project(mypkg LANGUAGES CXX)
set(CMAKE_CXX_STANDARD 20)

find_package(nanobind CONFIG REQUIRED)

nanobind_add_module(_mypkg src/bindings.cpp)
install(TARGETS _mypkg LIBRARY DESTINATION mypkg)

python/mypkg/__init__.py

from ._mypkg import add, squared, Counter

__all__ = ["add", "squared", "Counter"]

Build and publish

pip install build twine
python -m build --wheel
twine upload --repository testpypi dist/*.whl

# Users then:
pip install --index-url https://test.pypi.org/simple mypkg

With nanobind + Stable ABI, one wheel per platform (macos-arm64, linux-x86_64, …) covers all supported Python versions. Without Stable ABI, you build a matrix (3.12, 3.13, 3.14 × platforms) via cibuildwheel.

8. What PyTorch itself uses — the context

PyTorch’s own Python bindings are pybind11. This is why:

  • Every custom PyTorch op tutorial you’ll find uses pybind11.

  • torch/extension.h is a thin layer on top of pybind11.

  • You cannot mix nanobind and pybind11 in a single extension — they own the interpreter state differently.

So: for PyTorch extensions, pybind11. For standalone C++ libraries where you control the ABI, nanobind (if new).

9. What most people get wrong

  • They hold the GIL for a 5-second C++ function. Nobody else runs. The user files a bug titled “Python is single-threaded”. Always release for heavy work.

  • They touch py::objects inside a released GIL region. Segfault. Reacquire first.

  • They accept py::array_t<float> without c_style | forcecast and get failures when users pass float64 or F-contiguous arrays. Add the flags.

  • They copy NumPy arrays into std::vector to “convert to C++”. Use .request().ptr and stay zero-copy.

  • They ship setup.py with hardcoded compiler flags. Use scikit-build-core + CMake.

  • They mix pybind11 and nanobind in the same package. Pick one per extension.

  • They pick nanobind on Python 3.11. Compiles but you lose the Stable ABI advantage. If you must support 3.11, pybind11 is easier.

  • They don’t ship type stubs. Users’ IDEs show Any for every function. Generate .pyi files with stubgen -m _mypkg or write them by hand.

  • They forget the exception translation layer. A C++ std::runtime_error propagates as Python RuntimeError (pybind11 does this automatically for a subset of std:: exceptions). For your own C++ exception types, register a translator.

10. Practice exercises

  1. Write a pybind11 module that exposes one function sum_squared(x: np.ndarray) -> float. Zero-copy, releases the GIL, benchmarks 5x faster than a NumPy np.sum(x**2) on 100 M floats. Report the ratio.

  2. Add a Counter class as above. Add __getstate__/__setstate__ so it pickles. Round-trip through pickle.dumps/loads.

  3. Rewrite the same module in nanobind. Compare compile time (time cmake --build build) and .so size.

  4. Use nanobind::ndarray to accept a PyTorch tensor and NumPy array through the same function. Verify both work.

  5. Package with scikit-build-core. Publish to test-PyPI. Install into a fresh venv and run a smoke test.

Nav: ← 03 ONNX Runtime C++ · Next: 05 Arrow and data pipelines →