The Debugging Stack: gdb, lldb, ASan, Valgrind, rr¶

Here is the honest truth about C debugging: if you can’t drive a debugger, you cannot write serious C. printf gets you through toy problems. For anything real — a segfault in a threaded network service, a corrupt struct 40 stack frames deep, a heisenbug that only appears every 50th run — you need gdb or lldb, and you need sanitizers, and eventually you need record-replay. This file makes you competent at all of them.

On your Mac, lldb ships with Xcode Command Line Tools. gdb does not; getting it running on Apple Silicon involves code-signing dance and it’s flaky. On macOS use lldb. On Linux use gdb. They’re mostly interchangeable in muscle memory.

The Top 20 Commands (gdb / lldb parity)¶

Intent	gdb	lldb
Start the program	`run` / `r`	`run` / `r`
Run with args	`run arg1 arg2`	`run arg1 arg2` or `process launch -- arg1 arg2`
Set a breakpoint	`break main` / `b main`	`breakpoint set --name main` / `b main`
Break at line	`b file.c:42`	`b file.c:42`
Break on condition	`b file.c:42 if x > 5`	`b file.c:42 --condition 'x > 5'`
List breakpoints	`info breakpoints` / `i b`	`breakpoint list` / `br l`
Delete breakpoint	`delete 2` / `d 2`	`breakpoint delete 2` / `br del 2`
Continue	`continue` / `c`	`continue` / `c`
Step into	`step` / `s`	`step` / `s`
Step over	`next` / `n`	`next` / `n`
Step out	`finish`	`finish`
Instruction step	`stepi` / `si`	`si`
Backtrace	`backtrace` / `bt`	`bt` or `thread backtrace`
Select frame	`frame 3` / `f 3`	`frame select 3` / `f 3`
Up/down stack	`up` / `down`	`up` / `down`
Print variable	`print x` / `p x`	`print x` / `p x` or `expression x`
Print struct	`p *ptr`	`p *ptr`
Examine memory	`x/16xb ptr` (16 bytes hex)	`memory read --size 1 --format x --count 16 ptr` or `x/16xb ptr`
Watchpoint on var	`watch x`	`watchpoint set variable x`
Show locals/args	`info locals` / `info args`	`frame variable` / `fr v`
Disassemble	`disassemble`	`disassemble`
Attach to PID	`gdb -p 1234`	`lldb -p 1234`

Memorize the left column. The right one you look up. Ninety percent of debugging sessions use b, r, n, s, c, bt, p, finish.

The Debugging Loop¶

cc -g -O0 -fsanitize=address,undefined prog.c -o prog
lldb ./prog
(lldb) b some_function
(lldb) run
(lldb) bt              # where am I
(lldb) fr v            # what are the locals
(lldb) n               # step past this line
(lldb) p some_var      # inspect
(lldb) c               # keep going

Do this 20 times this month. It should become as automatic as git status.

gdb TUI Mode¶

gdb -tui (or Ctrl-x a inside gdb) gives you a split view: source above, gdb prompt below. Massively better than staring at line numbers. lldb has gui (typed at prompt) which is similar though less polished.

`.gdbinit` / `.lldbinit`¶

Drop in your $HOME:

# ~/.gdbinit
set history save on
set print pretty on
set pagination off

# ~/.lldbinit
settings set target.max-string-summary-length 4096
command alias ll frame variable

gdb-dashboard (a single Python script on GitHub, unmaintained but works) turns gdb into a live-updating multi-panel debugger. Optional; nice once you’ve internalized the commands.

Sanitizers vs Valgrind: The Apple Silicon Reality¶

This is where the seed curriculum needs a correction, and it’s the most important finding in this file.

Valgrind does not run natively on Apple Silicon macOS. As of mid-2026 the upstream Valgrind project supports x86_64 macOS up to Mojave-era only. The Louis Brunner fork (LouisBrunner/valgrind-macos) provides some macOS support but arm64 status on the current macOS releases is still marked as ~ or partial. Community verdict on r/C_Programming and Stack Overflow: do not fight this. Options:

Use AddressSanitizer + UBSan + LeakSanitizer instead. They catch ~90% of what Valgrind’s memcheck catches, run 5-20× faster, and are native on your Mac. This is the recommendation.
Run Valgrind in a Linux arm64 Docker container if you specifically need Valgrind’s tools (Callgrind for cycle-accurate profiling, Helgrind for race detection, Massif for heap profiling). Native arm64 Linux Valgrind works fine; emulated x86 is punishingly slow.
Use Apple’s leaks utility (ships with Xcode CLT) for a quick leak check: leaks --atExit -- ./prog. It’s not as thorough as ASan but it’s zero-setup.

Sanitizer defaults you should know:

# ASan reports off exit
ASAN_OPTIONS=detect_leaks=1:abort_on_error=1:halt_on_error=1 ./prog

# UBSan pretty output
UBSAN_OPTIONS=print_stacktrace=1:halt_on_error=1 ./prog

Note: on Apple Clang, LeakSanitizer support is spotty; the LLVM Discourse thread from 2025 documents cases where it reports unresolvable stack traces. If you need real leak detection on macOS, either use leaks or run under Linux. On Linux with mainline Clang, -fsanitize=address includes LSan and it just works.

The Sanitizer Verdict Table¶

Tool	Platform	Overhead	Catches
ASan	mac + linux native	2× CPU, 3× RAM	heap/stack/global OOB, UAF, double-free
UBSan	mac + linux native	~1.2×	signed overflow, null deref, misalignment, shift OOB
LSan	linux native, macOS partial	negligible	leaks at exit
TSan	mac + linux native	5-15×	data races (build separately, not with ASan)
MSan (Clang)	linux only	3×	reads of uninitialized memory
Valgrind memcheck	linux native, macOS via Docker	20-30×	leaks, UAF, uninit reads, more

Practical rule for you on your M-series Mac: dev build with -fsanitize=address,undefined. If you need TSan, rebuild with -fsanitize=thread separately. If a test suite passes ASan clean, you’re 90% of the way to memory-correctness.

rr: Record-Replay Debugging¶

rr is a Mozilla-built tool that records program execution once and then lets you replay it deterministically — including running the debugger backward. Set a watchpoint on a memory address, then reverse-continue to find who last wrote to it. This is transformative for heisenbugs.

Bad news: rr requires Linux and historically requires x86_64 CPU performance counters. So on your Apple Silicon Mac, native rr is not available.

Good news, as of 2024-2025:

rr.soft (sidkshatriya’s fork) adds a software-counters mode that works on aarch64 Linux and cloud VMs where hardware perf counters are locked down. If you run rr inside a Linux arm64 VM (UTM, Lima, OrbStack) or an arm64 Docker container on your Mac, rr.soft is your path in.
Warpspeed (Nick Gregory’s REcon 2023 work) is a macOS record-replay debugger. Research-quality, not a mainstream tool yet, but the trajectory is real.

For Phase 0 you don’t need rr. It’s a Phase 4+ tool for threading bugs. Just know it exists so future-you doesn’t waste four hours reproducing a race manually.

What Most People Get Wrong About Debugging¶

They printf-debug for far too long. The rationalization is “it’s faster to add a printf than fire up gdb.” That’s true for the first printf. It’s false by the third. Once you’re comfortable with b, r, n, p, c, bt, the debugger is always faster because you don’t have to rebuild between hypotheses, and you get real stack traces instead of guessed variable dumps. Every C programmer who never got past printf regrets it. Get over the hump this month.

The second thing they get wrong: they run debug builds without sanitizers, so they debug the wrong bug. ASan catches the first out-of-bounds write, not the symptom five function calls later. Fire your bug at ASan first — half the time you don’t need gdb at all because ASan tells you the exact file:line of the corruption.

Cheat Sheet¶

# Build for debugging
cc -g -O0 -fsanitize=address,undefined prog.c -o prog

# Reproduce a crash
./prog                # ASan/UBSan will print stack trace and abort

# Interactive debug
lldb ./prog            # or: gdb ./prog
(lldb) b main
(lldb) r arg1 arg2
(lldb) bt
(lldb) fr v
(lldb) n / s / c / finish

# Post-mortem from a core dump
ulimit -c unlimited    # enable core dumps
./prog                 # crash
lldb ./prog -c /cores/core.<pid>   # macOS default core path

# Attach to a running process
lldb -p $(pgrep prog)

# Leak check on macOS without Valgrind
leaks --atExit -- ./prog

# Docker for real Valgrind on arm64 mac (only if you need it)
docker run --rm -it --platform linux/arm64 -v $PWD:/w -w /w \
  ubuntu:24.04 bash -c "apt update && apt install -y build-essential valgrind && \
  cc -g prog.c -o prog && valgrind --leak-check=full ./prog"

Return to README.md · Next: 04_the_first_week_reset.md