Debugging Concurrent C

Concurrency bugs are qualitatively different from every other bug you’ve debugged. A memory leak reproduces on the next run. A segfault reproduces under gdb. A race condition happens once in ten thousand runs, corrupts state so mildly that the crash is thirty seconds later, and disappears the instant you attach a debugger because the debugger slows down one thread by a microsecond and the window closes. Every senior systems engineer has a war story about a bug that took a week to catch because they were using the wrong tools. This file exists so you don’t get to have one of those.

The short version: for concurrency bugs you go TSan first, gdb second, and always with a reproduction loop in a shell script. Never trust “I ran it and it worked.” Concurrency correctness is a claim you make with tooling, not with vibes.

ThreadSanitizer — the tool you use every day now

TSan (ThreadSanitizer) is a runtime race detector built into clang and gcc. It instruments every memory access and every synchronization primitive at compile time; at runtime it maintains a happens-before graph and flags any two accesses to the same memory location where at least one is a write and no happens-before edge orders them. That is the textbook definition of a data race, and TSan finds them with near-zero false positives.

$ cc -fsanitize=thread -g -O1 -o server_tsan server.c -lpthread
$ ./server_tsan &
$ wrk -c 200 -t 4 -d 30s http://localhost:8080/

TSan output looks like:

WARNING: ThreadSanitizer: data race (pid=<phone_number_or_numberic_id_or_random_id_33>)
  Write of size 8 at 0x7b0400001440 by thread T3:
    #0 accept_conn networking.c:142
  Previous read of size 8 at 0x7b0400001440 by thread T1:
    #0 stats_report metrics.c:88
  Location is heap block of size 4096 at 0x7b0400001000
  Thread T3 (tid=<phone_number_or_numberic_id_or_random_id_34>) created by main thread at:
    #0 pthread_create <null>
    #1 spawn_workers main.c:31

Read this carefully because it’s the entire debugging signal you need: which two lines, which two threads, and which byte of memory. That’s more information than you’d extract from a full day of printf archaeology.

The performance cost is real

• CPU: typical 5-15× slowdown. Sometimes closer to 3× on IO-bound workloads, sometimes 20× on tight compute loops. TSan’s own docs cite “5×-15× typical.”

• Memory: 5-10× higher RSS. TSan keeps shadow memory 8× the size of every allocation, plus the happens-before history.

• You cannot ship TSan-instrumented binaries to production. Run it under integration tests and load tests, then ship a clean build.

The 5-15× cost is what makes TSan practical: it’s slow enough to notice, fast enough to run in CI on a nightly integration job. Compare Valgrind’s helgrind at 20-50× and you understand why TSan replaced it for most day-to-day work.

When TSan lies

Rare but worth knowing:

• Custom synchronization primitives (a lock-free ring buffer with hand-rolled atomics) can confuse TSan into false positives if you didn’t use <stdatomic.h> atomics with proper memory orders. TSan reads C11 atomics correctly; it does not read your inline assembly. Annotate with __tsan_acquire() / __tsan_release() at ring push/pop if you must, or better: use <stdatomic.h> and let TSan see the acquire/release edges natively.

• File-descriptor sharing across threads without a lock. TSan sometimes doesn’t model kernel side effects; a write to a socket in thread A and a read in thread B looks race-free to TSan because the kernel is the ordering domain. This is correct behavior; just be aware that “TSan clean” doesn’t equal “no logical bugs.”

• Signal handlers. TSan can’t fully instrument signal handlers. Keep them minimal (write to a volatile sig_atomic_t flag, done).

Overall: treat TSan as the ground truth for data races on shared memory, and use other tools for the corner cases.

helgrind (Valgrind) — the fallback, but nearly dead on Apple Silicon

helgrind is Valgrind’s race detector: valgrind --tool=helgrind ./your_program. It works by intercepting pthreads calls and shadowing every load/store. On x86_64 Linux it’s fine, and it can catch some things TSan misses (particularly around uninstrumented library code). On Apple Silicon it’s essentially unusable: as of mid-2026 Valgrind on macOS/arm64 remains a maintenance-mode port with no working helgrind support for modern SDK versions. This point came up in Phase 4’s tooling review — stick to Linux (or a Linux VM) for concurrency work if you’re on a Mac.

The short verdict:

Tool	Cost	When
TSan	5-15×	Every commit, in CI
helgrind (Linux x86_64)	20-50×	When TSan misses something suspicious
helgrind (macOS arm64)	—	Effectively unavailable in 2026

Deadlock reproduction — the tight-loop-with-random-yield trick

A deadlock that shows up once every 10 000 runs will drive you insane. The move is to manufacture the interleaving.

// Insert into the middle of every lock-acquiring code path during
// deadlock hunts (compiled out with #ifdef DEBUG_RACE):
static inline void race_yield(void) {
    if ((rand() & 0xFF) == 0)
        sched_yield();     // or usleep(rand() % 100);
}

Call race_yield() between every lock_a and lock_b acquisition in the suspected paths. This inflates the chance of catching a lock-order inversion from 1-in-10 000 to 1-in-10 or better. Combine with a while true; do ./test; done shell loop and you’ll typically hang within seconds.

When it hangs, in a separate shell:

$ pstack $(pidof your_prog)     # Linux; or: gdb -p PID, then thread apply all bt

pstack gives you a stack trace of every thread. A deadlock is unmistakable: two threads both blocked in pthread_mutex_lock, at symmetric points in your code. Now you have a lock-order violation locked in on paper. Fix by imposing a global lock ordering.

Lock-ordering discipline

The only reliable way to prevent deadlock in code with more than one mutex: impose a total order on locks and always acquire in that order. In production codebases this is enforced by:

• Comments at every lock declaration stating its order rank.

• Assertions — assert(current_thread_max_lock_rank < this_lock.rank) before every acquisition, in debug builds. pthread_mutex_lock doesn’t do this for you; you build it on top with a thread-local counter.

• Never taking a lock while holding a lock of equal or higher rank.

If your code has three or more mutexes and no documented order, you have a latent deadlock. It’s just a matter of time.

Load testing tools — the 2026 landscape

You need load. Small toys don’t reveal concurrency bugs; production traffic patterns do. Here’s the honest state of the tools:

Tool	Language	HTTP versions	Sweet spot
wrk	C + LuaJIT	HTTP/1.1	Still the default for raw HTTP/1.1 throughput. Single binary, low overhead.
wrk2	C fork	HTTP/1.1	Constant-throughput mode (-R); the right tool for tail-latency (--latency gives correct HdrHistogram output, no coordinated omission)
vegeta	Go	HTTP/1.1, HTTP/2	Programmatic, target-URL-file driven, great for scripted mixed workloads. HTTP/2 support since 2018
h2load	C++ (nghttp2)	HTTP/1.1, HTTP/2, HTTP/3	The go-to for HTTP/2 and HTTP/3 (QUIC) benchmarking. Bundled with nghttp2.
oha	Rust	HTTP/1.1, HTTP/2	Modern, wrk-style CLI with a live TUI. Nice for quick local runs. Growing in adoption.
k6	Go + JS scripting	HTTP/1.1, HTTP/2, WebSocket	Scriptable in ES6, great for scenario tests, heavier than wrk. Grafana Labs.
hey	Go	HTTP/1.1	ab-alike, tiny, no reason to prefer over vegeta or wrk.
ab (ApacheBench)	C	HTTP/1.1	Legacy, single-threaded, poor tail-latency reporting. Ignore.

For your HARD GATE #1 (Rung 5 HTTP echo server):

• Primary tool: wrk. Command: wrk -c 1000 -t 4 -d 60s http://127.0.0.1:8080/. Single-binary install (brew install wrk on macOS, apt install wrk on Debian/Ubuntu). Reported target: 100k rps sustained.

• Tail-latency tool: wrk2. Command: wrk2 -c 1000 -t 4 -d 60s -R 100000 --latency http://127.0.0.1:8080/. The -R is fixed throughput; the --latency output uses HdrHistogram and correctly reports p99 and p99.9, unlike vanilla wrk which suffers from coordinated omission. Gil Tene’s talk “How NOT to measure latency” (Strange Loop 2013, YouTube) is the reference — watch it once.

• HTTP/2 stretch: h2load. If you add HTTP/2 support (projects.md stretch), h2load is the canonical benchmark tool.

wrk is still the standard raw-throughput tool in 2026 despite challengers, mostly because it’s a single 2-file C program you can trust, running LuaJIT for scripting when you need it. Newer tools (oha, k6, vegeta) each have their niche but haven’t dethroned wrk for the specific job of “push a C server to its throughput ceiling.”

A debug story — the stale-read race in the Rung 5 HTTP server

Here’s the shape of a bug you will hit. Setup: your epoll HTTP server has one acceptor thread and 4 worker threads. Workers pull ready fds from a shared queue and run the request lifecycle. You benchmark under wrk -c 500 -t 4 -d 60s, throughput is fine, everything looks good — until every ~<phone_number_or_numberic_id_or_random_id_36>th request returns garbage bytes.

Step 1: reproduce in a loop. Don’t trust one run. Wrap:

for i in $(seq 1 200); do
    ./server &
    SERVER=$!
    wrk -c 500 -t 4 -d 5s http://127.0.0.1:8080/ > /tmp/wrk.$i.log 2>&1
    curl -s http://127.0.0.1:8080/ | grep -c GARBAGE >> /tmp/garbage_count.log
    kill $SERVER; wait
done

Now you have 200 samples and a repeatable measurement of the failure rate.

Step 2: TSan. Rebuild with -fsanitize=thread -g -O1. Run the same loop with a smaller wrk -c 50 -t 4 -d 10s (TSan’s slowdown makes 500 conns painful). If TSan reports a race, you’re done — fix it and rerun.

Typical output for this bug:

WARNING: ThreadSanitizer: data race
  Write of size 4 at 0x7b04... by thread T3:
    #0 conn_reset_buffer conn.c:88
  Previous read of size 4 at 0x7b04... by thread T2:
    #0 conn_next_state conn.c:52

Read: thread T3 (worker) reset the connection buffer while thread T2 (another worker) was reading its state. Cause: two workers picked up events for the same fd. Fix: EPOLLONESHOT on client fds so only one worker sees a given fd until it re-arms.

Step 3: verify. Add EPOLLONESHOT, re-run the loop, confirm garbage count is zero across 200 runs. Now rebuild without TSan (-O2 -DNDEBUG) and rerun for throughput. If throughput matches the pre-fix number, ship it.

Step 4: keep the harness. That 200-iteration loop becomes a CI job. Every future change runs it before merge. Bugs you found once you find again.

The printf-with-thread-id habit

A one-liner that pays off for every concurrency debug session:

#include <sys/syscall.h>
#include <unistd.h>
#define TID ((int)syscall(SYS_gettid))
#define TLOG(fmt, ...) fprintf(stderr, "[T%d %s:%d] " fmt "\n", TID, __func__, __LINE__, ##__VA_ARGS__)

Sprinkle TLOG("accept fd=%d", cfd) at every state transition. When you see interleaved output like:

[T5 accept_conn:142] accept fd=17
[T3 worker_loop:88] pop fd=17
[T2 worker_loop:88] pop fd=17     ← wrong. two threads on same fd.
[T3 conn_read:52] read fd=17 n=45
[T2 conn_read:52] read fd=17 n=-1 (EAGAIN)

you see the bug in the log before you even attach a debugger. Cheap, effective, and completely portable across every Unix.

When to reach for gdb (rarely)

gdb is your tool for a hung program (deadlock, spinlock, infinite loop). Attach, thread apply all bt, read the stacks, done. It is not your tool for a race condition — the very act of running under gdb changes the timing enough to hide most races. Keep gdb for:

• Post-crash core dumps (segfaults, aborts).

• Deadlock reproduction (once you’ve hung with race_yield, attach and inspect).

• Watchpoints on specific memory locations (rare but occasionally the perfect tool).

What most people get wrong about this

They chase concurrency bugs in gdb alone. gdb is a fine tool for sequential bugs and a mediocre tool for concurrent ones — you can’t step through a race, and setting a breakpoint changes the timing. The right progression is: TSan first (nine times out of ten it finds the bug), then reproduce in a loop under wrk or wrk2 (never trust one run), then read the code with the TSan report as your map, and only then reach for gdb if the bug is a deadlock or a crash. Reversing that order costs days.

Second common mistake: shipping a fix without a regression harness. A race you found once is a race you’ll re-introduce in six months. Every concurrency bug you find deserves a scripted reproducer that lives in your test suite. “Runs clean under TSan with wrk -c 1000 -t 4 -d 60s” is a check-in-able criterion; “I fixed it, trust me” is not.

Practice this week

1. Take the thread pool from file 04. Introduce a deliberate race by removing the mutex around pending-- in the worker. Run under TSan. Confirm you get a clean report pointing at the exact line.

2. Take your echo server from file 05. Multi-thread the accept + read path badly (share a single accept fd across two threads with no EPOLLONESHOT). Under wrk -c 500 you should see garbled responses. Confirm TSan flags it.

3. Set up wrk2 -R 50000 --latency against your echo server. Capture the p50/p99/p99.9. Watch the Gil Tene “How NOT to measure latency” talk (26 min, YouTube) and confirm you understand why wrk2 with -R gives you correct numbers and vanilla wrk does not.

4. Write a run_concurrency_ci.sh script: builds with TSan, builds with ASan, runs each against wrk -c 200 -t 4 -d 30s, fails if either sanitizer reports anything. This is your Rung 5 acceptance harness.

5. Reproduce a deadlock: two mutexes, two threads, take them in opposite order in each thread with a usleep(1) in the middle. Confirm the program hangs, pstack shows the two mutex waiters, fix by imposing a lock order, confirm no hang.

References

• ThreadSanitizer manual — github.com/google/sanitizers/wiki/ThreadSanitizerCppManual. Small, complete.

• Gil Tene, “How NOT to measure latency” — Strange Loop 2013, YouTube. 26 minutes. Watch. Once.

• wrk repository — github.com/wg/wrk. The Lua scripting docs at SCRIPTING are worth 15 minutes.

• wrk2 repository — github.com/giltene/wrk2. Read the README for the coordinated-omission argument.

• Valgrind helgrind manual — valgrind.org/docs/manual/hg-manual.html. For x86_64 Linux fallback.

• Brendan Gregg, Systems Performance, 2nd ed., chapter 6 (CPUs) and chapter 10 (Networking). Carried over from Phase 4; used heavily here.

• Paul McKenney, Is Parallel Programming Hard — chapters 11 (“Validation”) and 12 (“Formal Verification”) are the deep dive on why concurrency testing is different in kind from sequential testing.

---

Return to README.md · Next: projects.md