06 — Profilers and Tracers¶

By M9 you must be able to run a C program under a profiler and answer “where is the time going?” without guessing. Guessing is what junior engineers do; measurement is what senior engineers do. This file is the toolchain. Every tool listed is free.

The 2026 landscape¶

Tool	Platform	Type	Overhead	Primary use
perf	Linux	Sampling profiler + counters	Low (1-3%)	CPU time, cache misses, branch mispredicts
hotspot	Linux	GUI for perf	Same as perf	Visualize perf data as flame graphs, call trees
Instruments	macOS	Sampling profiler (Xcode)	Low	The Mac equivalent of perf
Tracy	Cross-platform	Instrumented profiler	Low if you insert zones	Frame-by-frame, low-latency loops (game/audio/HFT)
bpftrace	Linux	Tracing (eBPF)	Variable	Dynamic tracing of syscalls, kernel events
ftrace	Linux	Kernel tracer	Low	Kernel function tracing
strace	Linux	syscall tracer	High (2-100x)	“What syscalls is this process making?”
dtrace	macOS + illumos	Dynamic tracing	Low	Historical; on macOS partially disabled by SIP
eu-perf-report / flamegraph.pl	Linux	Post-processing	—	Turn perf output into flame graphs

perf on Linux — the workhorse¶

Install: sudo apt install linux-tools-common linux-tools-$(uname -r) on Ubuntu. Some cloud VMs require additional setup.

Basic recipes:

# Where is CPU time going?
perf record -F 99 -g -- ./prog
perf report

# CPU cycles, cache misses, branch misses at once
perf stat -e cycles,instructions,cache-misses,cache-references,branch-misses ./prog

# Live top-like view
perf top

# Kernel and user together (needs root usually)
sudo perf record -F 99 -a -g -- sleep 10
sudo perf report

Turn perf output into a flame graph (Brendan Gregg’s classic):

git clone https://github.com/brendangregg/FlameGraph
perf record -F 99 -g -- ./prog
perf script | FlameGraph/stackcollapse-perf.pl | FlameGraph/flamegraph.pl > out.svg
open out.svg

Set /proc/sys/kernel/perf_event_paranoid to 1 (or -1) to allow unprivileged perf. Some distros default to 4 and you’ll see “No permission” errors otherwise.

hotspot — GUI for perf¶

KDE’s hotspot is a Qt-based GUI that wraps perf record and shows flame graphs, top-down/bottom-up trees, and disassembly annotated with samples. On Ubuntu: sudo apt install hotspot.

Worth using once you have a real profiling problem. Not needed on Day 1.

Instruments on macOS¶

Xcode’s Instruments is genuinely excellent — possibly the best sampling profiler UI on any platform. Launch: xcrun xctrace or just run Instruments.app (part of Xcode; requires full Xcode, not just Command Line Tools).

Useful templates for C:

Time Profiler — sampling CPU profile.
System Trace — syscalls and scheduler.
Allocations — heap allocation tracking (redundant with ASan for correctness; useful for perf).
Counters — CPU perf counters (Apple Silicon has its own set; documented at https://developer.apple.com/documentation/apple-silicon).

Command-line equivalent: sample <pid> 10 — sample any running process for 10 seconds, print a call-tree summary.

Tracy Profiler — for low-latency C¶

Tracy is a nanosecond-resolution frame profiler. You add small macros to your code marking zones:

#include <tracy/TracyC.h>

void render_frame(void) {
    TracyCZone(ctx, 1);
    // ... work ...
    TracyCZoneEnd(ctx);
}

A separate GUI client connects over TCP and shows a real-time timeline. Perfect for game engines, audio processing, HFT-style code, and “why is this loop 3x slower than expected” analysis.

Supports: C, C++, CUDA, Vulkan, D3D11/12, Metal. 10,500+ commits, actively maintained.

When to use: M11+ when you’re doing SIMD/perf-critical work. Overkill for general profiling.

bpftrace — dynamic tracing¶

bpftrace is eBPF-backed dynamic tracing with a DTrace-inspired language. Alive at https://github.com/bpftrace/bpftrace, FOSDEM 2025 talk. Linux 5.x+ only.

Example one-liners:

# Trace all open() syscalls from a process
sudo bpftrace -e 'tracepoint:syscalls:sys_enter_openat { printf("%s %s\n", comm, str(args->filename)); }'

# Histogram of read() sizes
sudo bpftrace -e 'tracepoint:syscalls:sys_enter_read { @sizes = hist(args->count); }'

When to reach for it: you have a running production system and want to attach a probe without stopping it. This is a M10+ skill.

Brendan Gregg’s book “BPF Performance Tools” is the canonical reference.

Latency numbers, memorized¶

Before any profiling, know the rough numbers. Peter Norvig’s original “Latency Numbers Every Programmer Should Know” (see 09_resources/05_papers_and_talks.md):

L1 cache reference               0.5 ns
Branch mispredict                5   ns
L2 cache reference               7   ns
Mutex lock/unlock                25  ns
Main memory reference            100 ns
Compress 1KB with zippy          3   μs
Send 1KB over 1 Gbps network     10  μs
Read 4KB randomly from SSD       150 μs
Read 1MB sequentially from RAM   250 μs
Round trip within data center    500 μs
Read 1MB sequentially from SSD   1   ms
Disk seek                        10  ms
Read 1MB sequentially from disk  20  ms
CA → Netherlands → CA network    150 ms

Memorize the order of magnitude. When you’re guessing where time is going, these numbers tell you if your guess is even plausible.

What most people get wrong about profiling¶

They guess. They add printf timers. They optimize the wrong loop. Rule: never optimize C without a profiler. Ever. If you “think it’s slow because of X,” measure. Then measure again after your change. Amdahl’s law is not a suggestion.

Return to README.md · Next: 07_containers_and_vms.md