04 — Profiling and Performance

Read this file twice. Then read it again after your first profiling session, because half of it will only make sense once you have stared at a flamegraph.

Profiling is the skill that separates “C++ engineers” from “C++ engineers companies pay to make their systems faster.” Most bootcamp graduates and most CS undergrads have never perf recorded a real binary in their lives. If you do it competently — predict a hotspot, measure it, fix it, prove the fix — you are already in the top decile of applicants for any C++ backend / ML infra role. This is the single highest-leverage file in Phase 6. Take it seriously.

The mindset shift: predict → measure → analyze delta

Untrained engineers optimize by intuition. “This loop looks slow, let me unroll it.” Trained engineers optimize by hypothesis:

  1. Predict. Before you profile, write down where you think the time is going. “I think 60% is in inference, 25% in JSON parsing, 15% in TLS.”

  2. Measure. Run the profiler. See where time actually goes.

  3. Analyze the delta. Where were you wrong? That gap is the most valuable output of the session. Your mental model of the system just improved.

  4. Change one thing. Not five. One.

  5. Re-measure. Prove the change did what you thought.

Skip step 1 and you are just a mechanic swapping parts. Include step 1 and you become a diagnostician. Every senior study partner can tell the difference in 90 seconds of conversation.

The two kinds of profilers you must know

Kind

How it works

Overhead

Answers

Sampling

Interrupts the process N times/sec, records call stack

1-3%

“What is expensive across the whole run?”

Instrumentation

Records enter/exit of every marked region

Varies (5-50%)

“How long did this specific block take, per call, timeline-aware?”

Sampling is perf, Instruments’ Time Profiler, VTune. Instrumentation is Tracy, -pg gprof (dead), callgrind. Modern practice uses both: sampling to find the region, instrumentation to zoom in on it.

Linux: perf — the swiss army knife

perf is a kernel-integrated sampling profiler. On any Linux box (including your $5 VM for P6.1) it is the default. Learn the four subcommands cold.

perf stat — the 20-second sanity check

perf stat -e task-clock,context-switches,cpu-migrations,page-faults,\
cycles,instructions,branches,branch-misses,\
L1-dcache-load-misses,LLC-load-misses ./miniserve

Gives you: IPC (instructions per cycle — target > 1.0, ideal > 2.0), cache miss rates, branch prediction rate. If IPC is 0.3, you are memory-bound and no algorithm tweak in your inner loop will help until you fix data layout.

perf record — the actual profile

perf record -F 999 -g --call-graph=dwarf ./miniserve --load-test
# ...ctrl-c after ~30s of representative load...
perf report --stdio | head -50

Flags to know: -F 999 = 999 Hz sampling (odd numbers avoid resonance with periodic timers), -g --call-graph=dwarf = capture full stack via DWARF unwinding (worth the size cost — frame-pointer stacks are often broken in optimized code).

perf annotate — hotspot at instruction level

perf annotate -s YourClass::hotFunction

Shows the assembly with sample counts per instruction. This is where you learn what your compiler actually did.

Flamegraphs — Brendan Gregg’s gift to humanity

perf record -F 999 -g -- ./miniserve
perf script > out.perf
./FlameGraph/stackcollapse-perf.pl out.perf > out.folded
./FlameGraph/flamegraph.pl out.folded > flame.svg

Open flame.svg in a browser. Width = time spent. The wide plateau at the top is your hotspot. This is the artifact you put in your P6.3 blog post — side-by-side before/after flamegraphs are the single most convincing thing you can show.

Get FlameGraph from github.com/brendangregg/FlameGraph. It is Perl. Do not overthink it.

macOS: Instruments (and its two useful tabs)

On your Apple Silicon MacBook, perf does not exist. You have Instruments, which is XCode’s profiler and is genuinely good.

The two tabs you use:

  • Time Profiler — statistical sampling, equivalent to perf record. Shows the flamegraph in the “heaviest stack trace” pane.

  • Allocations — tracks every malloc/free. Essential for finding leaks, temporary-allocation storms, and “why does this small function allocate?”

Launch: Instruments.app → pick Time Profiler → drag your binary in → Record. On Apple Silicon, Time Profiler leverages hardware counters via Xcode’s CoreProfile plumbing — the results are trustworthy.

One footgun: Instruments needs a release-with-symbols binary. Compile with -O2 -g -fno-omit-frame-pointer — not -O0, not Release stripped. Debug builds profile faster in absolute terms but their hot spots are unrelated to release hot spots. Do not waste hours profiling Debug and then “optimizing.”

Tracy: real-time, in-process, timeline

Tracy is the instrumentation profiler you actually enjoy using. It runs a client inside your binary, streams events to a GUI over TCP, and shows you a nanosecond-resolution timeline you can zoom into as your program is running. Adobe uses it in Photoshop. Trading firms use it. Game engines use it. You should use it in P6.3.

Setup: clone Tracy, add its TracyClient.cpp to your CMake, link, and sprinkle ZoneScoped; inside functions you care about:

#include <tracy/Tracy.hpp>

void predict(const Request& req) {
  ZoneScoped;
  ZoneText(req.model_name.c_str(), req.model_name.size());
  // ...
  {
    ZoneScopedN("json_parse");
    parse(req.body);
  }
  {
    ZoneScopedN("model_infer");
    run(req.features);
  }
}

Build the Tracy-release viewer (from the profiler subdir), run it, click “Connect” — your service pops up. Timeline shows every zone, per thread, with per-call durations, plots for user-defined values, memory allocation view, GPU zones if you add them.

Why Tracy over perf on Linux? Because perf gives you aggregate stats; Tracy shows you the timeline. “P99 spikes every 30 seconds” is invisible in perf and screaming at you in Tracy. If your P6.1 has periodic tail-latency issues, Tracy is how you find them.

Overhead: single-digit percent when idle, ~5-10% when actively recording zones. Fine for staging. Not for prod hot path unless you compile it out with -DTRACY_ENABLE=OFF.

Cachegrind and heaptrack: when you need cache/memory truth

  • valgrind --tool=cachegrind ./bin — simulates a cache hierarchy and counts hits/misses per line. Slow (10-40x). Deterministic. Use to answer “is this function cache-bound?” when perf’s LLC-miss counter is ambiguous.

  • heaptrack ./bin — KDE project. Traces every allocation, produces a per-function allocation flamegraph. Better than Instruments’ Allocations for headless Linux services.

Both are for targeted investigations, not daily use.

Micro vs macro benchmarks: when Google Benchmark misleads

Google Benchmark (the benchmark:: library) is excellent for measuring one function in isolation. It handles warmup, disables ASLR variance somewhat, computes CV. Use it for:

  • “Is std::unordered_map or absl::flat_hash_map faster for my keys?”

  • “Does SIMD help this pixel loop?”

  • “How much does moving from virtual to if/switch gain?”

Do not use it for:

  • “Is my gRPC service faster?” — microbenchmarks cannot see network, queue, or thread-pool effects.

  • “Did my allocator swap help?” — allocator wins show up under multi-thread contention, which microbenchmarks rarely reproduce.

  • “Is my p99 better?” — microbenchmarks measure mean/median; tail is a system property.

Rule: micro for algorithms, macro (load test → percentiles) for services. Do not brag about a 30% microbenchmark improvement without showing the end-to-end p99 moved.

The five bugs every profiling newbie hits

  1. Profiling a Debug build. -O0 code has completely different hot spots than -O2. Always profile release-with-symbols: -O2 -g -fno-omit-frame-pointer.

  2. Cold-cache measurements. First run is always slower. Discard it. Or use benchmark:: which warms up.

  3. Frequency scaling on. Modern CPUs throttle up and down. On Linux, sudo cpupower frequency-set -g performance before serious runs. On macOS, plug in the charger — battery mode throttles.

  4. No load, or wrong load. A profiler run at idle tells you your program can idle. Run under representative traffic — use ghz or k6 to generate it.

  5. Aggregating away the tail. Mean latency lies. Always look at p50/p95/p99/p99.9. A service with mean 5ms and p99 500ms is a broken service.

Concrete Phase 6 exercise: the profiling loop

Do this exact sequence in W41. It is the deliverable for that week.

  1. Pick one function in P6.1 you think is slow. Write down why in a note.

  2. Compile release-with-symbols. Deploy locally.

  3. Run ghz at 200 QPS for 60s against /predict. Capture p50/p99.

  4. In another terminal, perf record -F 999 -g for the middle 30s. Or Tracy if on Mac.

  5. Generate a flamegraph. Save before.svg.

  6. Read the flamegraph. Is your predicted hotspot actually the widest bar? Note the delta.

  7. Fix the actual hotspot. One change.

  8. Re-run steps 3–5. Save after.svg.

  9. Compute p99 improvement percentage.

  10. Write it up as your P6.3 blog post.

The first time you do this and get a >2x speedup by staring at a flamegraph, you will understand why senior engineers talk about profiling like it is a superpower. It is.

What most people get wrong

  • “Let me add -O3 and re-benchmark.” Compile flags are the least interesting axis. Layout, allocation, and algorithm changes are 10-100x more impactful.

  • Optimizing without a load generator. No load, no truth. Every serious profile is under representative traffic.

  • Reporting mean latency. Nobody cares about mean. p99 pays your salary.

  • Never reading the assembly. For your absolute hottest 10 lines of code, pop them into godbolt.org and read the disassembly. Otherwise you cannot explain why an intrinsic helped.

  • Trusting one measurement. Run each configuration 5+ times. Report median and CV. Modern hardware is noisy.

The reading list to accompany this file

Only four items. Read them all before W44.

  1. Brendan Gregg, Systems Performance (2nd ed, 2020) — the reference textbook. Chapters 6 (CPUs) and 13 (perf) are essential.

  2. Denis Bakhvalov, Performance Analysis and Tuning on Modern CPUs — free PDF, deeply practical, IPC/uArch focused.

  3. Google Benchmark docs — 40 pages. Read them.

  4. Bartek Filipek’s spdlog + Tracy series on bfilipek.com (2024–2025) — practical wiring, not theory.