The C++ Memory Model — Rigorously¶
“Sequential consistency is a lie the compiler tells you so you can sleep at night. Concurrent code is where the lie breaks.”
This is the single most important file in the entire roadmap. If you finish Phase 3 with only file 01 truly internalized, you will still be dangerous. If you skim it, no amount of std::mutex practice will save you.
Why a memory model exists¶
Modern CPUs and modern compilers both reorder your memory operations. The compiler does it to keep the pipeline fed; the CPU does it because loads and stores hit different caches, TLBs, and store buffers, and waiting for each to complete in program order would leave the ALU idle 90% of the time.
This reordering is fine for single-threaded code (the reordering is invisible to a single observer). It is catastrophic for multi-threaded code, because thread B may observe thread A’s writes in an order thread A never wrote them.
The memory model is the contract between the programmer, the compiler, and the CPU: it specifies exactly which reorderings are allowed and which are forbidden. C++11 was the first standard to include one. Before C++11, portable concurrent C++ was, strictly speaking, impossible.
Data race — the definition that matters¶
A data race occurs when two threads access the same memory location, at least one access is a write, and neither access happens-before the other.
Any data race is undefined behavior in C++. Not “produces the wrong answer” — UB. The compiler is licensed to do literally anything, including format your disk. In practice it produces mysterious, non-reproducible bugs that only appear under load. This is why the sanitizer for detecting races (TSan) is your best friend in Phase 3.
std::atomic<T> — the primitive¶
std::atomic<T> provides atomic read-modify-write operations on T, and a memory ordering argument that specifies how surrounding non-atomic operations are ordered relative to it.
#include <atomic>
std::atomic<int> counter{0};
counter.store(1, std::memory_order_relaxed);
int x = counter.load(std::memory_order_acquire);
counter.fetch_add(1, std::memory_order_acq_rel);
bool ok = counter.compare_exchange_weak(expected, desired,
std::memory_order_acq_rel,
std::memory_order_acquire);
Every atomic operation has a memory-order tag. Default is memory_order_seq_cst (strongest, and safest for beginners). All other orders are optimizations you use after you understand the model.
The six memory orders¶
Order |
Semantics |
When to use |
|---|---|---|
|
Atomic w.r.t. same location; no ordering with other memory |
Counters that only need atomicity, not ordering |
|
Ordered only with dependent loads |
Do not use. Deprecated in practice; every implementation upgrades it to |
|
This load synchronizes-with a |
Reading a shared flag / pointer |
|
This store synchronizes-with a matching |
Publishing a shared flag / pointer |
|
Both acquire and release for read-modify-write |
RMW on a mutex-like flag |
|
All above + all |
Default; safest; slowest on ARM |
Two rules of thumb. (1) When in doubt, use seq_cst — the performance cost on x86 is small; on ARM it emits dmb ish fences, which are not free but are correct. (2) Never lower to relaxed without understanding why. Every senior C++ engineer has a story about a colleague who “optimized” a seq_cst to relaxed and broke production.
The three canonical examples — memorize these¶
Example 1 — relaxed counter (correct)¶
std::atomic<uint64_t> hit_count{0};
// Any thread:
hit_count.fetch_add(1, std::memory_order_relaxed);
Why relaxed is enough. You only need atomicity, not any ordering with other data. If two threads increment simultaneously, both increments happen (no lost updates). You do not care about the order relative to other memory.
When relaxed is wrong. If any other thread reads hit_count and then reads other shared state that depends on it, you need acquire/release — the relaxed counter guarantees nothing about the state of other memory.
Example 2 — acquire / release for a message flag (the classic)¶
std::atomic<bool> ready{false};
int payload = 0; // NOT atomic
// Producer thread:
payload = 42; // (A)
ready.store(true, std::memory_order_release); // (B)
// Consumer thread:
while (!ready.load(std::memory_order_acquire)) {} // (C)
assert(payload == 42); // (D) — guaranteed
Why this works. The release store at (B) synchronizes-with the acquire load at (C) when the load sees the value written by the store. This synchronization creates a happens-before edge: (A) happens-before (B), (B) synchronizes-with (C), (C) happens-before (D). Therefore (A) happens-before (D), and the assert cannot fire.
Why relaxed here is broken. Without release/acquire, the compiler or CPU could reorder (A) and (B), or (C) and (D). The consumer might see ready == true but payload == 0. This is exactly what happens on ARM and almost never happens on x86 — which is why the bug ships from an x86 dev laptop.
Example 3 — seq_cst for global agreement (Dekker/Peterson)¶
std::atomic<bool> x{false}, y{false};
int r1 = 0, r2 = 0;
// Thread 1:
x.store(true, std::memory_order_seq_cst);
r1 = y.load(std::memory_order_seq_cst);
// Thread 2:
y.store(true, std::memory_order_seq_cst);
r2 = x.load(std::memory_order_seq_cst);
// After both threads finish, at least one of r1, r2 is true.
Why only seq_cst guarantees this. With acq_rel, x86’s store buffer can delay the store past the load, and on ARM the effect is even more visible. Both r1 and r2 can be false. Only seq_cst establishes a total order across all seq_cst operations everywhere, forbidding this outcome.
Cost on ARM. seq_cst loads and stores on ARM emit dmb ish (data memory barrier, inner shareable). This is measurable in benchmarks. If you do not need the total order, do not pay for it.
Happens-before and synchronizes-with — the two words that matter¶
Sequenced-before — within a single thread, one operation is sequenced-before another (roughly, program order).
Synchronizes-with — across threads, a release operation synchronizes-with the acquire operation that sees its value.
Happens-before — the transitive closure of sequenced-before and synchronizes-with.
The rule: if A happens-before B, then a write in A is visible to (and correctly ordered against) a read in B. If A does not happen-before B, no guarantees. If both threads access the same memory with no happens-before, and one is a write, it is a data race and it is UB.
Every threading question you will be asked reduces to: “draw the happens-before graph.” Do this on paper for Example 2 above until it is second nature.
Compiler vs CPU reordering¶
Both of them reorder. You must guard against both.
Compiler reordering¶
The compiler is a source-to-source transformation. It sees this:
int a = 1;
int b = 2;
…and freely re-emits it as b = 2; a = 1; if that is faster. For non-atomic accesses to non-shared memory, this is invisible and correct. For threaded code, only atomic operations (or a matching atomic_thread_fence) restrain the compiler.
CPU reordering — the store buffer and invalidation queue¶
Every modern CPU core has:
A store buffer: writes go here first before draining to the L1 cache. This means “store x=1; load y” can appear to other cores as “load y; store x=1” (StoreLoad reordering).
An invalidation queue: cache-line invalidations from other cores queue up; the local core may briefly read stale data before processing the queue.
On x86 (Intel/AMD/most desktops):
All-store-order is enforced (SPO / TSO). Stores are seen in program order by all cores.
Loads may still be reordered with earlier stores (StoreLoad) — the only reordering x86 allows.
mfenceor alock-prefixed instruction flushes the store buffer.
On ARM (Apple Silicon, Graviton, mobile):
Almost anything can reorder. Load-load, load-store, store-store, store-load — all can be observed out of order without a fence.
dmb ish(inner-shareable data memory barrier) is the primary ordering primitive.This is called the “weak” memory model. It is exactly why concurrent code that works on x86 often breaks on ARM.
On Apple Silicon specifically: M1/M2/M3/M4 have a special TSO mode used by Rosetta 2 (per Wrenger et al., 2024). Under TSO the chip behaves like x86 at ~9% throughput cost. Native ARM binaries — what your Xcode-compiled C++ actually runs as — do not use TSO. So your local tests can pass with the weak model, and the bugs surface only when contention is high enough to expose the reordering.
atomic_thread_fence — the standalone fence¶
Sometimes you need a fence without an associated atomic operation. std::atomic_thread_fence(std::memory_order_...) provides one.
std::atomic<bool> ready{false};
int payload = 0;
// Producer:
payload = 42;
std::atomic_thread_fence(std::memory_order_release);
ready.store(true, std::memory_order_relaxed);
// Consumer:
while (!ready.load(std::memory_order_relaxed)) {}
std::atomic_thread_fence(std::memory_order_acquire);
assert(payload == 42);
This is equivalent to the release/acquire pattern on ready itself, but the fence version is useful when you want to keep the atomic op cheap (relaxed) while still establishing the ordering elsewhere. Rarely needed in application code, common in lock-free library code (see file 04).
DCLP — double-checked locking, done wrong and done right¶
Pre-C++11, the “double-checked locking pattern” for lazy singleton init looked like this. It was broken.
// BROKEN pre-C++11. Do not use.
static Singleton* instance = nullptr;
static std::mutex m;
Singleton* get() {
if (instance == nullptr) { // (1) unsynchronized read
std::lock_guard<std::mutex> lk(m);
if (instance == nullptr) {
instance = new Singleton(); // (2) publish
}
}
return instance;
}
Why broken: the read at (1) may see a non-null pointer whose pointee is not yet fully constructed. The compiler may reorder instance = new Singleton() into instance = malloc(); construct(instance) — and another thread sees the pointer before construction finishes.
The C++11 fix using atomics:
static std::atomic<Singleton*> instance{nullptr};
static std::mutex m;
Singleton* get() {
Singleton* p = instance.load(std::memory_order_acquire);
if (p == nullptr) {
std::lock_guard<std::mutex> lk(m);
p = instance.load(std::memory_order_relaxed);
if (p == nullptr) {
p = new Singleton();
instance.store(p, std::memory_order_release);
}
}
return p;
}
The release store publishes a fully-constructed pointer; the acquire load ensures the reader sees the construction. This is DCLP done right, and is the canonical example of “why you should never write pre-C++11 concurrent code from muscle memory.”
The one-liner alternative. In modern C++, prefer:
static std::once_flag flag;
Singleton* p = nullptr;
std::call_once(flag, [&]{ p = new Singleton(); });
Or, even simpler and thread-safe since C++11:
Singleton& get() {
static Singleton s; // Meyers singleton; thread-safe init since C++11
return s;
}
Use the static-local. DCLP is an study question, not a production pattern in 2026.
Required reading (do this before file 02)¶
Herb Sutter, “atomic<> Weapons” (2013, both parts). Still the definitive practitioner talk. On YouTube. ~3 hours. Take notes.
Preshing on Programming. Jeff Preshing’s blog. Read at minimum: “Memory Ordering at Compile Time,” “An Introduction to Lock-Free Programming,” “Weak vs Strong Memory Models,” “Acquire and Release Semantics.”
Wrenger et al., 2024, Analyzing the memory ordering models of the Apple M1. ~20 pages. Read to understand TSO vs weak on your own machine.
cppreference.com,
std::memory_order. Read the whole page, twice.
Do not proceed to threads (file 02) until you can, on a blank page, draw the happens-before graph for Example 2 above and explain why relaxed breaks it.
What most people get wrong¶
They read this file, feel enlightened, and then two weeks later default to relaxed in their own code because they “know it is faster.” It is not faster in any meaningful sense for most code, and it is often wrong. When you are unsure, use seq_cst. The overhead of a dmb ish on ARM is measured in single-digit nanoseconds per operation. The overhead of a race condition in production is a Sev-1.
The second failure mode: they assume “my code works on my Mac” is meaningful evidence for correctness. It is not. Apple Silicon’s weak model can mask bugs at low contention because the reorderings are probabilistic. TSan is the tool. Run it. On every test. Every commit.
The third: they never internalize “synchronizes-with is between a specific store and a specific load that reads its value.” They think release/acquire is a global barrier. It is not. It is pairwise, per-location. Draw the pairs.
Nav: ← Phase 3 README · Threads & sync →