Thread Pools and Work Queues

One thread per request is the naive answer, and it’s the wrong one. Thread creation costs ~10-50 μs on Linux. Kernel stacks default to 8 MB of virtual memory per thread. Above a few thousand threads, context switching, TLB pressure, and scheduling overhead crush throughput. Every production server on Earth — nginx, redis, envoy, PostgreSQL, Triton — uses either a fixed pool of worker threads consuming a shared queue, an event loop (files 06-07), or a hybrid of the two.

This file walks you through building a proper thread pool from scratch. By the end you’ll have a header-only mini-library you can drop into any project, and you’ll understand why work-stealing (used by Rayon, Tokio, and the Java ForkJoinPool) is the next step up. The pool you build here is the exact same pattern used by the request-dispatch layer of Triton Inference Server — you’re not learning a toy.

The three components

Every thread pool has:

1. A task queue — a bounded or unbounded queue of (function, argument) pairs.

2. A worker pool — N threads, each in a loop: pop a task, run it, repeat.

3. A shutdown protocol — a way to tell the workers “drain and exit” without leaking tasks or hanging.

The queue from file 01 (bounded, mutex + two cond vars) is your starting point. What follows layers on top.

A minimal but correct thread pool

#include <pthread.h>
#include <stdlib.h>
#include <stdbool.h>

typedef struct task {
    void (*fn)(void *);
    void *arg;
    struct task *next;
} task_t;

typedef struct {
    pthread_mutex_t m;
    pthread_cond_t  has_work;
    pthread_cond_t  all_idle;         // for wait-until-drained
    task_t *head, *tail;
    int      pending;                 // queued + in-flight
    bool     shutdown;
    int      nthreads;
    pthread_t *threads;
} tpool_t;

static void *worker(void *arg) {
    tpool_t *p = arg;
    for (;;) {
        pthread_mutex_lock(&p->m);
        while (!p->head && !p->shutdown)
            pthread_cond_wait(&p->has_work, &p->m);
        if (p->shutdown && !p->head) {
            pthread_mutex_unlock(&p->m);
            return NULL;
        }
        task_t *t = p->head;
        p->head = t->next;
        if (!p->head) p->tail = NULL;
        pthread_mutex_unlock(&p->m);

        t->fn(t->arg);
        free(t);

        pthread_mutex_lock(&p->m);
        if (--p->pending == 0) pthread_cond_broadcast(&p->all_idle);
        pthread_mutex_unlock(&p->m);
    }
}

tpool_t *tpool_create(int n) {
    tpool_t *p = calloc(1, sizeof(*p));
    pthread_mutex_init(&p->m, NULL);
    pthread_cond_init(&p->has_work, NULL);
    pthread_cond_init(&p->all_idle, NULL);
    p->nthreads = n;
    p->threads = calloc(n, sizeof(pthread_t));
    for (int i = 0; i < n; i++)
        pthread_create(&p->threads[i], NULL, worker, p);
    return p;
}

void tpool_submit(tpool_t *p, void (*fn)(void*), void *arg) {
    task_t *t = malloc(sizeof(*t));
    t->fn = fn; t->arg = arg; t->next = NULL;
    pthread_mutex_lock(&p->m);
    if (p->tail) p->tail->next = t;
    else         p->head = t;
    p->tail = t;
    p->pending++;
    pthread_cond_signal(&p->has_work);
    pthread_mutex_unlock(&p->m);
}

void tpool_wait(tpool_t *p) {
    pthread_mutex_lock(&p->m);
    while (p->pending > 0) pthread_cond_wait(&p->all_idle, &p->m);
    pthread_mutex_unlock(&p->m);
}

void tpool_destroy(tpool_t *p) {
    pthread_mutex_lock(&p->m);
    p->shutdown = true;
    pthread_cond_broadcast(&p->has_work);
    pthread_mutex_unlock(&p->m);
    for (int i = 0; i < p->nthreads; i++) pthread_join(p->threads[i], NULL);
    // free residual queue if any (only reachable on abrupt shutdown paths)
    free(p->threads);
    pthread_mutex_destroy(&p->m);
    pthread_cond_destroy(&p->has_work);
    pthread_cond_destroy(&p->all_idle);
    free(p);
}

Read this pattern until it’s second nature. Every serious C server has a variant of it. Points to note:

• while (!p->head && !p->shutdown) — the wait-loop-while-condition idiom from file 01. Not negotiable.

• Shutdown is graceful: workers finish any task they’ve already dequeued, then exit when both the queue is empty and shutdown is set.

• tpool_wait uses a separate cond var (all_idle). Broadcasting to has_work would wake workers who then re-sleep; a dedicated cond var is cleaner.

• pending counts queued + in-flight. Only when both are zero is the pool truly quiescent.

• pthread_cond_signal on submit, pthread_cond_broadcast on shutdown. Signal wakes one; broadcast wakes all. On shutdown you want everyone up.

Sizing the pool

Rule of thumb:

• CPU-bound tasks: nthreads = number of physical cores (get from sysconf(_SC_NPROCESSORS_ONLN), adjusted for hyperthreading if benchmarks say so).

• I/O-bound tasks: more threads than cores helps because threads block on I/O. Numbers like 2-4× cores are common. Above that, gains flatten and context-switch cost dominates.

• Mixed: measure. Use two pools — a small CPU pool and a larger I/O pool.

In an inference server context: typically one pool for request preprocessing/postprocessing (I/O-bound, JSON parsing, tokenization) and a separate GPU worker (single-threaded, batches requests, dispatches to CUDA). Two pools, two disciplines.

Common bugs and their fixes

Bug	Symptom	Fix
Missing signal after enqueue	Workers sleep forever with tasks queued	Always signal (or broadcast) inside the same critical section as the enqueue
Signal before enqueue	Wakes worker, worker sees empty queue, re-sleeps, wasted wake	Enqueue then signal, atomically under the lock
Free task before running	Segfault mid-task	The worker owns the task once dequeued; free after t->fn returns
Broadcast every submit	Thundering herd, N-1 workers wake and go back to sleep	Use signal, not broadcast, for single-task wakes
No shutdown flag check	Workers hang in cond_wait forever after destroy	The wait-loop must check both head and shutdown
Task returns before args released	Use-after-free	Whoever allocates the arg is responsible for freeing it; document convention

Work-stealing — the next level

A fixed queue is a scalability bottleneck at high thread counts: every submit and every dequeue serializes on p->m. Work-stealing solves this by giving each worker its own local deque; when a worker’s deque is empty, it steals from another worker’s deque.

• Rayon (Rust), Tokio (Rust async runtime), Java ForkJoinPool, Go’s scheduler, and Intel TBB are all work-stealing.

• The canonical algorithm is Chase-Lev (2005). Each worker’s deque is a lock-free structure where the owner pushes/pops from one end (“bottom”) and thieves steal from the other (“top”).

• This is genuinely hard to implement correctly — ABA problems, memory reclamation, all of it. For this phase, know the name and the intuition. For production, use libck or a similar library.

When do you need work-stealing? Only when profiling shows queue lock contention is your bottleneck. For most workloads at up to ~16 threads, a single fixed queue is fine. Above that, or for embarrassingly-parallel recursive tasks (Rayon’s sweet spot), work-stealing becomes worth the complexity.

Why event loops beat both for network I/O

The thread pool above is optimal when tasks are CPU work of similar duration. It’s the wrong tool when tasks are “read from a socket that might have data ready in 10 ms or 10 minutes.” A thread blocked in read() is doing nothing useful — you paid the thread’s memory and TLB cost for a sleep. At 10k concurrent connections this collapses.

The network answer is an event loop (files 06-07): one thread (or a small pool) driving epoll_wait, dispatching ready events without ever blocking on I/O. Combine event loop for I/O + thread pool for CPU work and you have the architecture of every high-performance server in existence.

We cover the event loop starting in file 06. For now, understand the split.

What most people get wrong about this

They grow the pool when it’s slow. “My server is slow — let me make the pool 200 threads.” This makes it worse: more threads compete for the same cores, contention on the queue lock increases, context switches multiply. The right response to a slow pool is: (a) profile individual tasks, (b) check if tasks are blocking on I/O that should be async, (c) check for lock contention inside tasks, (d) then consider more threads. Bigger is not faster.

Practice this week

1. Type the pool above from scratch. Ship it as tpool.h + tpool.c with a small test harness that submits 1M no-op tasks and verifies all run.

2. Add a tpool_submit_batch that takes an array of tasks and enqueues them under one lock acquisition. Measure throughput improvement for 100k task submits.

3. Build a small parallel wc (word count): submit one task per file in a directory tree, each task returns a count, aggregate in the main thread. Compare 1, 2, 4, 8 workers on your machine.

4. Under load (16 workers, tiny tasks), profile with perf and confirm that the queue’s mutex is the hot spot. This is the setup where work-stealing would help.

5. Read the source of a real thread pool: pthread_pool is a clean C reference (~300 LOC). Compare to yours.

References

• The pool structure above is the classic “boss-worker” pattern; Butenhof ch. 4 documents it in detail.

• Robert D. Blumofe, Charles E. Leiserson, “Scheduling Multithreaded Computations by Work Stealing” (1999) — the founding work-stealing paper. Readable.

• David Chase, Yossi Lev, “Dynamic Circular Work-Stealing Deque” (2005) — the deque algorithm most systems use.

• Rayon (Rust) internals docs — the clearest modern implementation writeup.

---

Return to README.md · Next: 05_sockets_from_scratch.md