03 — Testing and Benchmarks

A C++ project without a test suite and a benchmark suite is a personal science project, not an engineering artifact. GoogleTest is the industry default; Catch2 v3 is the elegant alternative you should be able to read; Google Benchmark is the only widely-accepted microbenchmark framework in the C++ ecosystem; nanobench is worth knowing when you want ten lines instead of a full CMake integration. This file walks each with the exact patterns you’ll actually use.

You need three things to leave this file: (1) a real gtest suite with fixtures, parameterized tests, mocks, and gtest_discover_tests wiring; (2) one working Catch2 v3 file so you know both APIs on sight; (3) a benchmark harness that measures what you meant to measure — no dead-code elimination, no cache-warmed-up-once-then-measure-something-else foot-guns.

1. GoogleTest — the workhorse

GoogleTest (aka gtest) is Google’s testing framework. It has been the C++ default since ~2010 and remains dominant in 2026. It ships gmock in the same repo. It integrates cleanly with CMake via FetchContent or find_package. It has good tooling support (IDE runners, JUnit XML output, TAP output).

Minimal setup (FetchContent route — easiest for a template repo)

include(FetchContent)
FetchContent_Declare(
    googletest
    GIT_REPOSITORY https://github.com/google/googletest.git
    GIT_TAG        v1.15.2)
set(gtest_force_shared_crt ON CACHE BOOL "" FORCE)
FetchContent_MakeAvailable(googletest)

enable_testing()
add_executable(mynn_tests
    tests/test_layer.cpp
    tests/test_train.cpp)
target_link_libraries(mynn_tests PRIVATE mynn GTest::gtest_main GTest::gmock)

include(GoogleTest)
gtest_discover_tests(mynn_tests)   # NOT add_test(...) — discover is better

gtest_discover_tests runs your binary once at build time, extracts the list of TEST(...)s, and registers each as a separate CTest entry. Result: ctest -j parallelizes per-test, and CI dashboards can show individual failures. Never use plain add_test(NAME mynn_tests COMMAND mynn_tests) — that’s one CTest entry that hides everything inside.

The five patterns you actually need

a) Basic test

#include <gtest/gtest.h>
#include "mynn/layer.hpp"

TEST(LinearLayer, ForwardShapeCorrect) {
    mynn::LinearLayer layer(/*in=*/4, /*out=*/3);
    Eigen::MatrixXd x = Eigen::MatrixXd::Random(2, 4);
    auto y = layer.forward(x);
    EXPECT_EQ(y.rows(), 2);
    EXPECT_EQ(y.cols(), 3);
}

EXPECT_* continues on failure; ASSERT_* aborts the test. Use ASSERT_* when a later line would crash if the current one fails (null pointer, dimension mismatch), otherwise EXPECT_*.

b) Fixture — shared setup for a set of tests

class LinearLayerTest : public ::testing::Test {
protected:
    void SetUp() override {
        layer_ = std::make_unique<mynn::LinearLayer>(4, 3);
    }
    std::unique_ptr<mynn::LinearLayer> layer_;
};

TEST_F(LinearLayerTest, WeightsInitialized) {
    EXPECT_EQ(layer_->weights().rows(), 3);
}
TEST_F(LinearLayerTest, BiasInitialized) {
    EXPECT_EQ(layer_->bias().size(), 3);
}

TEST_F (F for Fixture) creates a fresh instance per test. SetUp runs before each, TearDown after.

c) Parameterized test — same logic, many inputs

class ForwardShape : public ::testing::TestWithParam<std::tuple<int,int,int>> {};

TEST_P(ForwardShape, Correct) {
    auto [batch, in, out] = GetParam();
    mynn::LinearLayer layer(in, out);
    auto y = layer.forward(Eigen::MatrixXd::Random(batch, in));
    EXPECT_EQ(y.rows(), batch);
    EXPECT_EQ(y.cols(), out);
}

INSTANTIATE_TEST_SUITE_P(All, ForwardShape,
    ::testing::Values(
        std::make_tuple(1,  4,   3),
        std::make_tuple(32, 784, 128),
        std::make_tuple(64, 128, 10)));

d) Death test — assert that code aborts under a bad input

TEST(LinearLayerDeath, ShapeMismatchAborts) {
    mynn::LinearLayer layer(4, 3);
    EXPECT_DEATH({
        layer.forward(Eigen::MatrixXd::Random(2, 999));  // wrong input dim
    }, "dimension mismatch");
}

Death tests fork a subprocess and match a regex against stderr. Use for assert, std::terminate, or explicit abort(). Do not overuse — they’re slow.

e) gmock — mock an interface for isolated unit tests

#include <gmock/gmock.h>

class IOptimizer {
public:
    virtual ~IOptimizer() = default;
    virtual void step(mynn::Layer&, const Eigen::MatrixXd& grad) = 0;
};

class MockOptimizer : public IOptimizer {
public:
    MOCK_METHOD(void, step, (mynn::Layer&, const Eigen::MatrixXd&), (override));
};

using ::testing::_;

TEST(Trainer, CallsStepOncePerBatch) {
    MockOptimizer opt;
    EXPECT_CALL(opt, step(_, _)).Times(1);
    mynn::Trainer trainer(opt);
    trainer.train_batch(/*x=*/..., /*y=*/...);
}

2. Catch2 v3 — the modern alternative

Catch2 v3 (major rewrite, released 2022, stable through 2026) split into precompiled library form. It’s macro-heavier but nicer to read; it shines for BDD-style specs.

Setup

FetchContent_Declare(
    Catch2
    GIT_REPOSITORY https://github.com/catchorg/Catch2.git
    GIT_TAG        v3.7.1)
FetchContent_MakeAvailable(Catch2)

add_executable(mynn_catch_tests tests/test_layer_catch.cpp)
target_link_libraries(mynn_catch_tests PRIVATE mynn Catch2::Catch2WithMain)

include(CTest)
include(Catch)
catch_discover_tests(mynn_catch_tests)   # gtest_discover_tests equivalent

Style

#include <catch2/catch_test_macros.hpp>
#include <catch2/matchers/catch_matchers_floating_point.hpp>

TEST_CASE("LinearLayer forward shape", "[layer]") {
    mynn::LinearLayer layer(4, 3);

    SECTION("batch of 2") {
        auto y = layer.forward(Eigen::MatrixXd::Random(2, 4));
        REQUIRE(y.rows() == 2);
        REQUIRE(y.cols() == 3);
    }
    SECTION("batch of 32") {
        auto y = layer.forward(Eigen::MatrixXd::Random(32, 4));
        REQUIRE(y.rows() == 32);
    }
}

SECTIONs re-run the outer setup independently — no fixture class needed. Tags in [...] let you filter (./tests "[layer]"). Matchers give expressive assertions:

REQUIRE_THAT(loss, Catch::Matchers::WithinAbs(0.5, 1e-6));

Which to pick

Use GoogleTest by default. It’s what employers use, what CI templates expect, what gmock ships with, what LLM assistants generate first. Learn Catch2 v3 so you can read and modify tests in projects that chose it. Do not port between them without a reason.

3. Google Benchmark — microbenchmarks that don’t lie

Microbenchmarks are famous for lying to you. The compiler eliminates “unused” results. The cache is hot from the prior iteration. The system scheduler moves you to a different core. Google Benchmark handles all of this if you use it correctly.

Setup

FetchContent_Declare(
    benchmark
    GIT_REPOSITORY https://github.com/google/benchmark.git
    GIT_TAG        v1.9.0)
set(BENCHMARK_ENABLE_TESTING OFF CACHE BOOL "" FORCE)  # or gtest fights
FetchContent_MakeAvailable(benchmark)

add_executable(mynn_bench benchmarks/bench_layer.cpp)
target_link_libraries(mynn_bench PRIVATE mynn benchmark::benchmark)

Always build benchmarks in Release or RelWithDebInfo. Debug numbers are meaningless.

The critical idioms

#include <benchmark/benchmark.h>
#include "mynn/layer.hpp"

static void BM_LinearForward(benchmark::State& state) {
    const int batch = state.range(0);
    mynn::LinearLayer layer(784, 128);
    auto x = Eigen::MatrixXd::Random(batch, 784);

    for (auto _ : state) {                    // the timed loop
        auto y = layer.forward(x);
        benchmark::DoNotOptimize(y);          // prevent compiler removing 'y'
        benchmark::ClobberMemory();           // force memory writes to be observable
    }
    state.SetItemsProcessed(state.iterations() * batch);
    state.SetBytesProcessed(state.iterations() * batch * 784 * sizeof(double));
}
BENCHMARK(BM_LinearForward)->Arg(1)->Arg(32)->Arg(256)->Arg(1024);

BENCHMARK_MAIN();

Two calls that matter:

  • benchmark::DoNotOptimize(x) — tells the compiler “assume x is observed by something you can’t see”. Without it, the optimizer sees y is never used and deletes the entire body of forward().

  • benchmark::ClobberMemory() — acts as a memory-barrier. Forces all writes-so-far to memory before the next iteration begins.

Forget these and your benchmark reports 0.5 ns/op for functions that do actual work. This is the classic microbenchmark lie.

Reading the output

BM_LinearForward/1        1240 ns    1240 ns   564020    items_per_second=806.451k/s
BM_LinearForward/32      12800 ns   12798 ns    54600    items_per_second=2.5M/s
BM_LinearForward/256     98400 ns   98380 ns     7100    items_per_second=2.6M/s
BM_LinearForward/1024   395000 ns  394800 ns     1770    items_per_second=2.59M/s

Columns:

  • Time: wall time per iteration (_/32 = batch size 32).

  • CPU: CPU time (should be ~= Time for non-multithreaded code).

  • Iterations: how many times Benchmark auto-ran the loop to get stable numbers.

  • items_per_second: from your SetItemsProcessed. This is the metric that transfers to a slide.

Throughput plateaus around batch=256 above — memory bandwidth is the bottleneck now, not the FLOPs. Reading benchmarks means reading these curves, not the single number.

Templated / typed benchmarks

template <typename T>
static void BM_MatMul(benchmark::State& state) {
    const int n = state.range(0);
    Eigen::Matrix<T, -1, -1> a = Eigen::Matrix<T, -1, -1>::Random(n, n);
    Eigen::Matrix<T, -1, -1> b = Eigen::Matrix<T, -1, -1>::Random(n, n);
    for (auto _ : state) {
        Eigen::Matrix<T, -1, -1> c = a * b;
        benchmark::DoNotOptimize(c);
    }
}
BENCHMARK(BM_MatMul<float>)->RangeMultiplier(2)->Range(64, 512);
BENCHMARK(BM_MatMul<double>)->RangeMultiplier(2)->Range(64, 512);

JSON output for CI regressions

./mynn_bench --benchmark_format=json --benchmark_out=bench_current.json

Compare with previous run:

python -m pip install google-benchmark
compare.py benchmarks bench_previous.json bench_current.json

Wire this into GHA to catch performance regressions.

4. nanobench — the ten-line alternative

When you’re prototyping in a scratch file and don’t want a full CMake target:

#include <nanobench.h>

int main() {
    ankerl::nanobench::Bench().run("linear forward b=32", [&] {
        auto y = layer.forward(x);
        ankerl::nanobench::doNotOptimizeAway(y);
    });
}

Single header. Prints a Markdown table you can paste into a README. Not a replacement for Google Benchmark in a serious project, but great for scratch measurements.

5. What most people get wrong

  • They benchmark in Debug builds. Numbers are 10–100x off. Always Release/RelWithDebInfo, -O2 or -O3, -march=native where legal.

  • They forget DoNotOptimize. The compiler deletes the entire benchmark body. They report absurd speedups.

  • They put timing code inline (std::chrono around a for-loop). The framework already knows how many iterations to run to hit statistical significance. Just use for (auto _ : state).

  • They benchmark once and trust the number. Run 3–5 times, take median, watch variance. Repeat on a quiet machine (close browser, disable turbo boost, pin to a core with taskset).

  • They mix tests and benchmarks in one binary. Separate targets. Tests run in CI on every PR. Benchmarks run nightly or on-demand.

  • They test only the happy path. No death tests, no fuzz inputs, no negative shapes. Then the ML code silently accepts a (0, 4) tensor and returns NaN.

  • They skip gtest_discover_tests. Then CI dashboards can’t show which specific test failed.

6. Practice exercises

  1. Take your Phase 2 or Phase 3 code. Write 10 gtest tests including at least one fixture, one parameterized, one death test.

  2. Rewrite two of those tests in Catch2 v3. Note which style you prefer.

  3. Write a Google Benchmark for one hot function. Use DoNotOptimize. Run at 4–5 input sizes. Report items_per_second.

  4. Deliberately remove DoNotOptimize. Watch the benchmark report 0.5 ns/op. Understand why.

Nav: ← 02 Conan and vcpkg · Next: 04 Sanitizers and static analysis →