Performance Intro: perf, flamegraphs, cachegrind, eBPF

Every performance discussion you’ve had so far has been theoretical: “cache-friendly access patterns,” “minimise allocations,” “avoid virtual functions.” This file is where you learn to measure, because on modern hardware your intuitions are wrong more often than they’re right. The 2×-slower code is often 2× faster than the version you “optimised” because you didn’t account for prefetching, branch prediction, or the TLB. Measurement is the only way out.

The path in 2026: perf for wall-clock and CPU counters, flamegraphs for shape, cachegrind for cache behaviour on small programs, eBPF (bpftrace/bcc) for anything running in production. In that order.

perf — the Linux profiling swiss army knife

perf is part of the Linux kernel source tree. It uses hardware performance counters and sampling to give you a picture of where your program spent its time. Install with linux-tools-common / linux-perf on Debian/Ubuntu.

The five commands you’ll use 90% of the time:

perf stat ./myprog                         # summary: cycles, instructions, cache-miss %, branch-miss %
perf top                                   # live system-wide sampling, like `top` for functions
perf record -F 99 --call-graph dwarf ./myprog
perf report                                # interactive browsable report of the above
perf script                                # raw stack traces — feed this into flamegraph.pl

Reading perf stat

 12,345,678,901   cycles                   #    3.21 GHz
  9,876,543,210   instructions             #    0.80  insn per cycle
     45,678,901   cache-misses             #   12.3% of all cache refs
     12,345,678   branch-misses            #    1.4% of all branches

  • IPC (instructions per cycle) < 1 typically means you’re memory-bound (waiting on cache).

  • IPC > 2 means you’re doing real compute; further wins need algorithm changes.

  • cache-miss % > 5% on a hot loop is a red flag — rethink data layout.

  • branch-miss % > 5% on a hot loop suggests unpredictable branches; consider branchless code, sorting, or __builtin_expect.

These anchors are for CPU-bound C on x86-64. For an ML inference kernel that’s expected to be memory-bound, IPC of 0.5 is normal and fine.

perf record and call graphs

perf record -F 99 --call-graph dwarf ./myprog
perf report --stdio

-F 99 samples at 99 Hz (odd numbers avoid aliasing with periodic system activity). --call-graph dwarf uses DWARF debug info for accurate stack unwinding — make sure you compiled with -g -fno-omit-frame-pointer. If your program is short, use -p <pid> on a longer run instead.

Prerequisite: compile with -g and often -fno-omit-frame-pointer. Without frame pointers, perf can still unwind via DWARF, but at higher cost. For production profiling, most large shops (Netflix, Meta) build with -fno-omit-frame-pointer specifically for this reason. Ubuntu 24.04 turned on frame pointers globally after Fedora led the way.

Flamegraphs — the shape of your program

Brendan Gregg’s flamegraph.pl (github.com/brendangregg/FlameGraph) turns perf script output into a stacked SVG where:

  • Y axis is stack depth (bottom = main, top = leaf functions).

  • X axis is sample count, not time (functions are alphabetically sorted per level).

  • Width of a bar = fraction of total samples in that function — wider means more time spent there.

perf record -F 99 --call-graph dwarf ./myprog
perf script | ./stackcollapse-perf.pl | ./flamegraph.pl > out.svg
firefox out.svg

Read it top-down: the flat plateaus at the top are where your program actually spent time. If you see read taking half the width, you’re I/O bound. If you see your own matmul function — congratulations, you have a computable-on optimization target.

Differential flamegraphs compare two runs (before/after a change). Look for red widening / green narrowing to see what regressed or improved. Invaluable during optimisation work.

cachegrind (valgrind) — slow but simulated

cachegrind runs your program under valgrind’s cache simulator. It gives you:

  • L1 instruction cache misses per line of code.

  • L1 data cache misses per line of code.

  • Last-level cache (L3) misses.

  • Branch mispredictions (with --branch-sim=yes).

valgrind --tool=cachegrind ./myprog
cg_annotate cachegrind.out.<pid>

Cost: 20-100× slower than native. Only usable on small programs or short test cases.

When to reach for it: you have a hot function and perf can’t tell you which line within it is missing cache. Cachegrind pinpoints. For anything larger than a few seconds of real work, use perf mem or eBPF-based tools instead.

A better and faster friend: callgrind (also part of valgrind) for call-count profiling with kcachegrind for a beautiful GUI. Useful when you don’t have symbols in production and can’t run perf.

perf mem and perf c2c — the memory tour

For cache-miss and false-sharing debugging on native runs (much faster than cachegrind):

perf mem record ./myprog
perf mem report
perf c2c record ./myprog                # cache-to-cache: catches false sharing across cores
perf c2c report

perf c2c in particular is one of those tools that seems obscure until you have a threading performance regression and it points straight at the two threads hammering adjacent cache lines. Worth knowing about; you’ll use it in Phase 5.

eBPF and bpftrace — the observability upgrade

We introduced eBPF in file 05. For performance work specifically, the tools you’ll want:

  • execsnoop — every exec() on the system.

  • opensnoop — every open() on the system.

  • biolatency — histogram of block I/O latencies.

  • tcpconnect, tcpaccept, tcplife — network connection tracing.

  • funclatency — histogram of latency for any kernel or user function.

  • profile — sampling CPU profiler as a bpftrace program.

Install bpfcc-tools (Debian/Ubuntu) or use the bcc-tools package on Fedora. All the tools above are one-liners.

When perf is not enough and eBPF is: production. eBPF is designed to have negligible overhead when tracing is off and low overhead when on. You can (and Netflix does) leave eBPF-based collectors running continuously. You cannot leave perf record running continuously.

When to reach for eBPF from Day 1: you have a bug that only reproduces in production, you can’t stop-the-world profile it, and you want a histogram of some specific event (“how long does our postgres query take, bucketed by 10ms?”).

The optimisation workflow — in one paragraph

  1. Get a repeatable benchmark. Without one, everything else lies.

  2. perf stat -r 10 ./bench for a stable summary (10 runs, mean/stddev).

  3. perf record + flamegraph to find where time is spent.

  4. Zoom into the top function. Look at IPC, cache misses, branch misses.

  5. Form a hypothesis. Change one thing.

  6. Rerun perf stat -r 10. Did the metric you targeted improve? Did the wall time improve?

  7. If yes, commit. If no, revert and think again.

Do not skip steps 1 and 6. “I optimised this function” without a repeatable benchmark and before/after numbers is not engineering; it’s decoration.

What most people get wrong about this

They optimise without profiling. They read a blog post that says “branches are slow” and rewrite readable code into a branchless mess that’s the same speed — or worse, because they broke the branch predictor’s happy path. Or they “parallelise for performance” a workload that turned out to be memory-bound, so 4 threads produce 1.1× speedup and a lot more cache pressure. Measure first. Change one thing. Measure again. This discipline is worth more than knowing every intrinsic on x86-64.

Practice this week

  1. Take your read_full/write_full from file 01. Write a benchmark that reads a 1GB file. Run under perf stat -r 5. Note IPC and cache-miss %.

  2. Write a matmul: C = A * B for 512×512 float matrices. Three versions: naive (ijk), transposed-B (ikj), blocked (32×32 tiles). Benchmark each. perf stat each. You will see IPC and L1 miss rates change dramatically. This is the same lesson every ML kernel writer learns.

  3. Generate a flamegraph of any program you’ve written that runs >5 seconds. Actually look at it. Identify one thing that surprises you.

  4. Install bpfcc-tools. Run execsnoop in one terminal while doing normal shell work in another. Notice how many processes actually get spawned.

  5. Read Brendan Gregg’s “Linux Perf Examples” page (brendangregg.com/perf.html) once through. Bookmark it.

References

  • Brendan Gregg, Systems Performance, 2nd ed. (2020) — the reference. Chapters 6, 12, 15 cover CPU/observability/eBPF respectively.

  • Brendan Gregg’s website (brendangregg.com) — the single best free performance-engineering resource on the internet.

  • Denis Bakhvalov, Performance Analysis and Tuning on Modern CPUs — free PDF, dense on hardware counters.

  • man perf, man perf-stat, man perf-record, man perf-report.

  • Julia Evans, “perf zine” — friendly quick-ref for perf subcommands.

Return to README.md · Next: projects.md