05 — Linux Syscalls for C++ Engineers

Every C++ abstraction eventually bottoms out at a syscall. std::mutex becomes futex on Linux. std::filesystem::copy becomes read/write (or copy_file_range if you’re lucky). std::thread becomes clone. When you profile a hot path and the flame graph is 40% kernel time, you cannot debug it without knowing which syscalls your library is calling and why.

You are on macOS Apple Silicon. That means two things: (1) you will use kqueue locally where Linux docs say epoll, and (2) most Zoho servers and every FAANG study whiteboard assume Linux. Learn both surfaces; keep the mapping in your head.

---

1. The mental model: file descriptors are the universal handle

On Unix, almost everything is a file descriptor: files, sockets, pipes, timers (timerfd), signals (signalfd), event objects (eventfd), even other epoll instances. That means one API — read/write/close plus a readiness notifier — handles all I/O. Master that and you understand 80% of a Linux server.

macOS keeps the same abstraction (fds are POSIX) but replaces the Linux-specific readiness syscalls with BSD equivalents.

2. mmap — the fastest way to touch a file

mmap maps a file (or anonymous memory) into your process’s address space. Reads and writes become memory accesses, dispatched by the page fault handler.

#include <sys/mman.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <unistd.h>

int fd = ::open("data.bin", O_RDONLY);
struct stat st{}; ::fstat(fd, &st);

void* p = ::mmap(nullptr, st.st_size, PROT_READ, MAP_PRIVATE, fd, 0);
if (p == MAP_FAILED) { /* handle */ }

// Sequential scan hint (Linux only; macOS ignores or partial support)
::madvise(p, st.st_size, MADV_SEQUENTIAL);

// Use it like a byte array:
const char* bytes = static_cast<const char*>(p);
// … parse …

::munmap(p, st.st_size);
::close(fd);

When to use mmap:

• Random access into a large file (parsing indexes, memory-mapped databases).

• Zero-copy sharing between processes (MAP_SHARED + anonymous file or shm_open).

• Startup speed: mmap an entire binary blob and let the page cache handle it.

When NOT to use it:

• Streaming a huge file top-to-bottom. read() with a small buffer is often faster because the kernel prefetches predictably; page faults on mmap are per-4KB and can thrash.

• Networked filesystems (NFS, SMB). Behavior is undefined-ish and hangs are common.

• Writing where you need fsync guarantees. msync semantics are subtle; prefer explicit write + fsync.

macOS note: mmap works identically. madvise flags differ; MADV_SEQUENTIAL and MADV_RANDOM exist, MADV_DONTNEED behaves differently (it actually frees pages on Linux; on macOS it’s advisory).

3. epoll (Linux) and kqueue (macOS/BSD) — the same idea, two APIs

You have N sockets. You want to know which ones are ready to read without spawning N threads or spinning on select. Both APIs solve this by registering interest once and receiving batched readiness events.

Linux epoll skeleton

#include <sys/epoll.h>
#include <unistd.h>

int ep = ::epoll_create1(EPOLL_CLOEXEC);

epoll_event ev{};
ev.events  = EPOLLIN | EPOLLET;   // edge-triggered
ev.data.fd = listen_fd;
::epoll_ctl(ep, EPOLL_CTL_ADD, listen_fd, &ev);

epoll_event events[64];
for (;;) {
    int n = ::epoll_wait(ep, events, 64, /*timeout_ms*/ -1);
    for (int i = 0; i < n; ++i) {
        int fd = events[i].data.fd;
        // read until EAGAIN because we're edge-triggered
        for (;;) {
            char buf[4096];
            ssize_t r = ::read(fd, buf, sizeof(buf));
            if (r > 0) { /* consume */ continue; }
            if (r == 0) { ::close(fd); break; }              // peer closed
            if (errno == EAGAIN || errno == EWOULDBLOCK) break;
            if (errno == EINTR) continue;
            /* real error */ ::close(fd); break;
        }
    }
}

Level-triggered vs edge-triggered (EPOLLET):

• Level: epoll_wait keeps returning the fd as long as it has data. Simpler; wastes wakeups on partial reads.

• Edge: notification fires once per state transition. You must drain to EAGAIN on every event or you will lose data. Fewer syscalls; harder to get right.

macOS kqueue skeleton (what you’ll actually run for P3.3)

#include <sys/event.h>
#include <sys/time.h>
#include <unistd.h>

int kq = ::kqueue();

struct kevent change{};
EV_SET(&change, listen_fd, EVFILT_READ, EV_ADD | EV_CLEAR, 0, 0, nullptr);
::kevent(kq, &change, 1, nullptr, 0, nullptr);

struct kevent events[64];
for (;;) {
    int n = ::kevent(kq, nullptr, 0, events, 64, /*timeout*/ nullptr);
    for (int i = 0; i < n; ++i) {
        int fd = (int)events[i].ident;
        if (events[i].filter == EVFILT_READ) {
            for (;;) {
                char buf[4096];
                ssize_t r = ::read(fd, buf, sizeof(buf));
                if (r > 0) continue;
                if (r == 0) { ::close(fd); break; }
                if (errno == EAGAIN || errno == EWOULDBLOCK) break;
                if (errno == EINTR) continue;
                ::close(fd); break;
            }
        }
    }
}

EV_CLEAR on kqueue is roughly EPOLLET on epoll — edge-triggered semantics.

Mapping table for your notes:

Concept	Linux	macOS
Readiness multiplexer	epoll	kqueue
Add/remove interest	epoll_ctl	kevent (change list)
Wait for events	epoll_wait	kevent (event list)
Edge trigger flag	EPOLLET	EV_CLEAR
Cross-fd timer	timerfd_create	EVFILT_TIMER
Cross-fd signal	signalfd	EVFILT_SIGNAL
User wakeup fd	eventfd	EVFILT_USER

Portability layer: if you want one API, use libraries — libuv, libevent, asio, mio. For P3.3 write kqueue directly and add three sentences of Linux-parity notes in your README.

What most people get wrong: they choose edge-triggered because a blog told them it was faster, then forget to loop until EAGAIN. Under load, half their bytes vanish and they can’t reproduce it in a debugger. Start with level-triggered. Move to edge-triggered only after a benchmark says it matters, and only after you’ve written the while(read == EAGAIN) loop.

4. futex — what std::mutex is on Linux

Linux std::mutex is a thin wrapper over futex(FUTEX_WAIT / FUTEX_WAKE). A futex is “an atomic integer in userspace with a syscall to sleep when it has a specific value.” Uncontended locks are userspace atomic ops; contention drops into the kernel.

You will not call futex() directly in production C++ (that’s what std::mutex is for), but reading the manpage (man 2 futex) is a one-hour investment that clarifies the mental model. On macOS the equivalent primitive is os_unfair_lock and (privately) __ulock_wait. std::mutex on modern macOS uses os_unfair_lock for uncontended fast paths as of libc++ recent versions; older versions used pthread over Mach ports and were noticeably slower.

5. clock_gettime — the only clock you should use

std::chrono::steady_clock on Linux and macOS both wrap clock_gettime. Pick the right clock ID for your purpose:

Clock ID	Guarantees	Use for
CLOCK_MONOTONIC	Never goes backward; not affected by NTP jumps	Timeouts, benchmarks, deadlines
CLOCK_MONOTONIC_RAW (Linux)	Not adjusted by NTP frequency slew either	Micro-benchmarks under time discipline
CLOCK_REALTIME	Wall clock; can jump	Timestamps for logs / disk
CLOCK_PROCESS_CPUTIME_ID	CPU time for this process	CPU cost isolation
CLOCK_THREAD_CPUTIME_ID	CPU time for this thread	Same, per-thread

On Apple Silicon mach_absolute_time is what backs the standard chrono clocks. It runs off the Apple architected 24 MHz timer — resolution is ~41 ns. Do not chase nanoseconds finer than that; you’re measuring rounding, not signal.

6. perf_event_open — for when the profiler isn’t answering

Linux perf_event_open gives you access to CPU performance counters (cycles, instructions, cache misses, branch mispredicts) from userspace. This is what perf stat uses under the hood. It is not something you call directly often — you use perf on the command line or google-benchmark’s --benchmark_perf_counters. Know that it exists so you can read the flame graphs.

macOS equivalent: Instruments.app and xctrace record --template 'CPU Counters'. The M-series exposes counters via the kperf framework; Apple doesn’t publish a stable userspace API, but Instruments works well enough for most performance work. For CPU-bound C++ hot-path analysis, use Instruments’ “CPU Profiler” and “System Trace” templates. For serious cycle counting, boot a Linux VM (or a qemu-system-aarch64 on M-series) and use perf.

7. Signals — one paragraph so you don’t get burned

Signals in a multithreaded C++ program are a minefield. The rules you need:

• Only async-signal-safe functions in a signal handler. That excludes printf, malloc, std::mutex, essentially everything you’d want.

• Block signals in worker threads and dedicate one thread to sigwait or signalfd. This is the “signal handling thread” pattern; it turns signals into normal fd events.

• Never install a SIGSEGV handler for control flow. Use it only to dump state and re-raise.

Portable rule: convert signals to fd events (signalfd on Linux, EVFILT_SIGNAL on macOS) and handle them in your event loop. No handlers.

8. strace (Linux) / dtruss (macOS) — the syscall X-ray

When behavior is inexplicable:

# Linux
strace -f -e trace=network,ipc -o trace.log ./your_program

# macOS (requires SIP disable for third-party binaries; your own works)
sudo dtruss -f ./your_program 2>&1 | head -200

Add these to your muscle memory. Any time a program “hangs,” “is slow,” or “doesn’t do what I said,” a syscall trace collapses the mystery in seconds.

9. Common bugs

1. EINTR unhandled. Every blocking syscall can return -1 with errno == EINTR if a signal fired. Loop.

2. Reading less than requested from a socket and assuming that’s an error. read on a socket can return short. Loop until you have what you need or hit EOF.

3. Closing an fd from thread A while thread B is poll/epoll_wait on it. Undefined. Use EVFILT_USER / eventfd to wake the loop first.

4. mmap-ing a file and then truncating it. Accessing the vanished pages is a SIGBUS, not EFAULT. Guard with sigaction or don’t do it.

5. Blocking accept on a listening socket after the process has forked — the parent and child both wake up on the same fd (thundering herd). Use SO_REUSEPORT and one accept per thread, or a single accept-thread.

6. Assuming Linux behavior on macOS. MSG_NOSIGNAL doesn’t exist (use SO_NOSIGPIPE on the socket or MSG_NOSIGNAL conditionally). epoll_* functions are absent. Read the mapping table.

---

Required reading

• man 2 mmap, man 2 epoll, man 2 kqueue, man 2 futex, man 7 signal-safety. The manpages are the source of truth. Read them once; skim them again when they bite you.

• Michael Kerrisk — The Linux Programming Interface. The reference book. Chapters 44 (pipes), 45 (SysV IPC), 63 (multiplexing), 44 (mmap). Skim table of contents, deep-read what you use.

• Julia Evans — Bite Size Linux / zines. Warm intro that stays honest about complexity. wizardzines.com.

• libuv design docs (docs.libuv.org/en/v1.x/design.html). Shows what a serious cross-platform event loop looks like inside.

• Beej’s Guide to Network Programming — free, still the friendliest intro to sockets.

Exercises

1. Write a cat clone in ~50 lines that open → mmap → write(STDOUT, …) → munmap. Time it vs /bin/cat on a 1 GB file. Explain the result to yourself.

2. Write a program that catches SIGINT via signalfd (or EVFILT_SIGNAL) and prints “goodbye” in the main loop, not in a handler. No signal() calls.

3. Run strace -c ls -l /usr/bin > /dev/null (or sudo dtruss -c) and identify the top-3 syscalls. Guess why before checking. Now do it on your own project’s binary.

---

Nav: ← 04 Lock-Free Basics · Phase 3 README · → 06 Undefined Behavior