epoll and io_uring

epoll is the concrete API behind every high-performance Linux server you’ve ever used. redis, nginx, envoy, HAProxy, Node.js (via libuv), Netty (Java NIO on Linux), Go’s netpoller — all epoll under the hood. It has been the right answer on Linux since kernel 2.6 (2004) and remains the right default answer in 2026. io_uring is the newer, batchier, fancier successor that shines on high-op-count workloads but comes with real caveats: kernel version dependency, disabled-in-Docker-by-default, and a much steeper learning curve. This file teaches epoll properly, then teaches you enough io_uring to make an informed choice for your Rung 5 project.

By the end you should be able to type an edge-triggered epoll loop correctly from memory, explain the read-until-EAGAIN discipline, and articulate why you’re using epoll for the HARD GATE HTTP server and not io_uring — with the security-and-deployment evidence to back it up.

The epoll API in five calls

int epoll_create1(int flags);
int epoll_ctl(int epfd, int op, int fd, struct epoll_event *event);
int epoll_wait(int epfd, struct epoll_event *events, int maxevents, int timeout);
// plus close(epfd) to tear down, and epoll_pwait if you need atomic signal masking

That’s it. Compare to the ~15 primitives in io_uring or the 20 in kqueue; epoll’s surface area is small on purpose.

Creation

int epfd = epoll_create1(EPOLL_CLOEXEC);
if (epfd < 0) err(1, "epoll_create1");

EPOLL_CLOEXEC sets close-on-exec so a fork+exec child doesn’t inherit your epoll fd. Always pass it. epoll_create1(0) works but forgetting CLOEXEC is a subtle security/leak bug you avoid by making it the default.

Registration

struct epoll_event ev = {
    .events = EPOLLIN | EPOLLET | EPOLLRDHUP,
    .data.fd = client_fd,     // or .data.ptr = your_conn_struct
};
epoll_ctl(epfd, EPOLL_CTL_ADD, client_fd, &ev);

Three ops: EPOLL_CTL_ADD, EPOLL_CTL_MOD, EPOLL_CTL_DEL. ADD on an already-registered fd returns EEXIST; MOD on an unregistered fd returns ENOENT. data is opaque to the kernel; production code puts a pointer to the per-connection struct there, so you get O(1) event → connection lookup with no hash table.

Waiting

struct epoll_event evs[64];
int n = epoll_wait(epfd, evs, 64, /*timeout_ms=*/-1);
for (int i = 0; i < n; i++) {
    conn_t *c = evs[i].data.ptr;
    uint32_t e = evs[i].events;
    if (e & EPOLLIN)   read_until_eagain(c);
    if (e & EPOLLOUT)  drain_write_queue(c);
    if (e & (EPOLLHUP | EPOLLERR | EPOLLRDHUP)) close_conn(c);
}

timeout is milliseconds; -1 blocks indefinitely, 0 polls, positive values time-limit the wait (used to drive per-loop tick work like timeouts). maxevents bounds how many events the kernel returns per call; typical values are 64-512. Above that you get diminishing returns and larger stack allocations.

Edge-triggered (ET) vs Level-triggered (LT)

This is the single most important thing to internalize about epoll.

Level-triggered (default): epoll_wait returns as long as the fd is currently readable/writable. If you read only half the buffered data and loop back, epoll_wait will return the fd again immediately. This matches poll() semantics and is easy to reason about.

Edge-triggered (EPOLLET): epoll_wait returns once, on the transition from not-readable to readable (or not-writable to writable). If you fail to drain the socket completely, the kernel will not tell you about the leftover data — you’ll stall until more data arrives. This is why every ET user must read until EAGAIN:

for (;;) {
    ssize_t n = read(fd, buf, sizeof buf);
    if (n > 0)          feed_parser(buf, n);
    else if (n == 0)    { peer_closed(fd); break; }
    else if (errno == EINTR)  continue;
    else if (errno == EAGAIN) break;      // drained; wait for next edge
    else                { close_err(fd); break; }
}

Why ET? Fewer wakeups — the kernel doesn’t re-notify you about data you already know exists. On a high-connection, high-op-count server this is measurable: at 100k connections doing tiny reads, LT gives you a wakeup for every readable moment; ET gives you a wakeup only when new data arrives. Production servers (nginx, redis, envoy) universally use ET plus read_until_eagain because the wakeup savings compound.

The three bugs everyone hits with ET:

1. Forgetting the drain loop. You read() once, get 512 bytes, return to the loop. The socket has 4 KB more buffered. epoll_wait doesn’t return the fd until new data arrives. Your connection appears to hang. Fix: always loop until EAGAIN.

2. Forgetting EPOLLET when you thought you set it. Copy-paste error. Test your code by running under strace and confirming the epoll_ctl call has EPOLLET in the flags.

3. Multi-threaded ET without EPOLLONESHOT. If two worker threads are both blocked on the same epoll_wait and a fd becomes readable, both threads may wake up on the same fd in ET mode (spurious wakeups exist even for ET — kernel doesn’t guarantee exactly-once semantics under multi-thread wait). Two threads calling read() on the same fd race. Fix: use EPOLLONESHOT, which auto-removes the fd from the epoll set after one wakeup; the handling thread must EPOLL_CTL_MOD to re-arm before returning to the loop. This is exactly nginx’s per-worker discipline.

The event flags you’ll actually use

Flag	Meaning
EPOLLIN	Readable (data available or peer closed)
EPOLLOUT	Writable (send buffer has space)
EPOLLRDHUP	Peer closed the write side (half-close). Better than waiting for a zero-length read
EPOLLHUP	Full hang-up. Set by kernel; you can’t request it
EPOLLERR	Error condition. Set by kernel; always monitor
EPOLLET	Edge-triggered mode
EPOLLONESHOT	Auto-disarm after one event; you must MOD to re-arm
EPOLLEXCLUSIVE	For thundering-herd on shared listen sockets; wake only one waiter

EPOLLEXCLUSIVE (Linux 4.5, 2016) is worth calling out for accept loops: if you have multiple threads each calling epoll_wait on the same listen fd, without EPOLLEXCLUSIVE all of them wake on a new connection and all but one go back to sleep after failing accept4. With EPOLLEXCLUSIVE, only one thread wakes. That said, the SO_REUSEPORT pattern (one epoll+listen socket per worker, kernel-hashed) is usually cleaner — that’s what nginx does now.

io_uring — the newer answer

io_uring was designed by Jens Axboe (Linux block layer maintainer) and merged in kernel 5.1 (May 2019). The core idea is shared-memory ring buffers between userspace and kernel, so you can submit many I/O operations without any syscall per op, and reap completions the same way.

The mental model

• Submission Queue (SQ): a ring of struct io_uring_sqe (Submission Queue Entries). You fill in an SQE (“read 4 KB from fd 12 into this buffer”) and advance the tail pointer.

• Completion Queue (CQ): a ring of struct io_uring_cqe (Completion Queue Entries). The kernel fills these in when ops finish. You reap them and advance the head pointer.

• io_uring_enter: the single syscall that hands submitted SQEs to the kernel and optionally waits for CQEs. With SQPOLL mode (a kernel thread polls the SQ ring), you don’t even need this syscall in steady state — zero-syscall I/O.

Raw io_uring is unpleasant to use directly (bit-fiddling ring indices, memory barriers). Everyone uses liburing (git clone git://git.kernel.dk/liburing) which wraps it in a sane C API:

struct io_uring ring;
io_uring_queue_init(256, &ring, 0);

struct io_uring_sqe *sqe = io_uring_get_sqe(&ring);
io_uring_prep_read(sqe, fd, buf, sizeof buf, /*offset=*/0);
io_uring_sqe_set_data(sqe, my_context_ptr);
io_uring_submit(&ring);

struct io_uring_cqe *cqe;
io_uring_wait_cqe(&ring, &cqe);
void *ctx = io_uring_cqe_get_data(cqe);
int bytes = cqe->res;                    // may be negative errno
io_uring_cqe_seen(&ring, cqe);

That’s the shape. A real server prepares many SQEs, submits in one call, waits for a batch of CQEs, dispatches them to per-connection state machines, repeats.

Kernel version matrix (verified July 2026)

Kernel	io_uring status
< 5.1	Not present
5.1 - 5.5	Present but with rough edges and many bugs; do not use for production
5.6+	“Basic” stability; the core ops (read/write/accept/connect/close) are solid
5.11+	fast poll, direct descriptors, most Rust/Tokio io_uring backends require this
6.0+	Multi-shot ops, ZC send, ring-mapped buffers — the “stable” fast-path target
6.6+ (LTS)	Current baseline for io_uring benchmarks in 2026

Ubuntu 24.04 LTS ships 6.8; Debian 13 (2025) ships 6.12; RHEL 9.x ships 5.14 with backports. If your target deploy is a modern LTS, you have io_uring. If your target is RHEL 8 (5.14 backported? no — 4.18) you do not.

The 2026 security-and-deployment reality (verified)

io_uring’s rich attack surface has produced a steady stream of CVEs. Public status as of mid-2026:

• Docker default seccomp profile blocks io_uring. The moby/moby issue #47532 is the canonical reference; docker/containerd began blocking io_uring_setup, io_uring_enter, and io_uring_register around 2023 after too many kernel CVEs, and the block is still in the default profile as of Docker 27+. Practical impact: if your service runs in a stock Docker container, io_uring will fail at startup with EPERM. You can opt back in with --security-opt seccomp=unconfined or a custom profile, but that widens your attack surface.

• Google Cloud disables io_uring on Cloud Run, App Engine, and older ChromeOS. Their public engineering blog cited “70% of Linux kernel exploits submitted to their VRP in one quarter came via io_uring.” That posture has not softened.

• Cloudflare disabled io_uring on production edge boxes after internal review; Marek Majkowski’s writeup on the Cloudflare blog (2022) is a widely-cited reference. Their argument: the API is a foothold for CVE-2026-46315-style local privilege escalation exploits with weak mitigation surface.

• Ongoing 2024-2026 CVE stream: CVE-2026-46315 (info-disclosure in io_uring/waitid, 2024), CVE-2026-43121 (io_uring zcrx freelist OOB write, 2025-2026), plus the io-wq exit race patched in early 2026. The frequency has slowed but not stopped.

Bottom line for you: io_uring is legitimate technology, actively maintained, benchmarks well, and is fine for bare-metal servers or trusted VMs. If your target is containers, managed cloud, or shared multi-tenant hosts, default to epoll. This is the same conclusion Cloudflare and Google reached with much bigger teams than you have.

io_uring vs epoll performance in 2026

Honest numbers from public benchmarks (Axboe’s own perf posts, tokio-uring benchmarks, ScyllaDB engineering blog):

• Simple HTTP echo, one connection at a time: io_uring and epoll are within 10% of each other. Both are dominated by TCP stack costs.

• High connection count + tiny ops (10k conns, 100+ ops each per loop): io_uring wins by 20-40% because it batches syscalls. This is io_uring’s sweet spot.

• Storage I/O (reading many files in parallel): io_uring can be 2-3× faster than the equivalent readv + threadpool because storage ops don’t have equivalents in epoll at all.

• Registered files + registered buffers + SQPOLL: approaches kernel-bypass numbers; ScyllaDB reports 1M+ ops/sec/core.

For your Rung 5 HTTP echo target of 100k rps: epoll is more than enough. You can do the io_uring variant as a stretch project (listed in projects.md) to compare and internalize the differences.

What most people get wrong about this

They assume io_uring is free performance and reach for it first. It isn’t and you shouldn’t. The API is genuinely harder — SQEs, CQEs, ring indices, multi-shot vs one-shot ops, buffer registration, SQPOLL, IORING_SETUP_COOP_TASKRUN — and the deployment reality (Docker, managed cloud) is hostile in a way that isn’t obvious from benchmark blog posts. Every ML inference server maintainer I know of who evaluated io_uring in 2023-2025 ended up back on epoll for the tier-1 code path, keeping io_uring for offline batch tools where the deployment surface is controlled. Do epoll first, benchmark, then decide.

Second mistake: level-triggered epoll with a single read() per event, no drain loop. This “works” for low-throughput demos and silently corrupts on the day traffic doubles. Always ET + read-until-EAGAIN.

Practice this week

1. Extend the echo server from file 05: switch to non-blocking sockets, add an epoll loop, handle 10k concurrent clients. Use ET + EPOLLRDHUP, drain to EAGAIN in one place. First correctness target, then benchmark with wrk -c 1000 -t 4 -d 30s.

2. Bug-hunt exercise: add a single read() call inside your ET handler (no loop), keep everything else. Verify with wrk -c 200 -R 5000 -d 60s that some connections stall. Then fix and re-benchmark.

3. Install liburing (apt install liburing-dev or build from git.kernel.dk/liburing). Type Axboe’s canonical cat example (SQE for read, print CQE res, done). Confirm it works on your kernel.

4. Try to run your liburing cat inside a stock Docker container with the default seccomp profile. Confirm io_uring_setup fails with EPERM. Now run with --security-opt seccomp=unconfined and confirm it works. This is the deployment reality made concrete.

5. Read redis/src/ae_epoll.c in full (~200 lines). Note how tiny the epoll wrapper is once you understand the primitives. Preview for file 08.

References

• man 7 epoll — read it end-to-end once. The kernel man page is unusually good for this API.

• Jens Axboe, “Efficient IO with io_uring” — the design paper. PDF at kernel.dk/io_uring.pdf. 30 pages, worth reading if you’ll ever use io_uring in anger.

• liburing repository — git.kernel.dk/liburing — examples/ directory is the tutorial.

• Marek Majkowski, Cloudflare, “Missing Manuals — io_uring worker pool” — cloudflare blog, 2022. The security stance is later in the series.

• moby/moby issue #47532 — the definitive source on Docker’s default seccomp policy for io_uring.

• LWN articles on io_uring — lwn.net/Kernel/Index/#io_uring — the ongoing chronicle of the API’s evolution. Jonathan Corbet’s coverage is unmatched.

• ScyllaDB engineering blog, “How io_uring and eBPF Will Revolutionize Programming in Linux” — benchmarks-heavy, biased toward io_uring; read critically.

---

Return to README.md · Next: 08_a_production_server_walkthrough.md