02 · SIMD & Vectorization¶

SIMD — Single Instruction Multiple Data — is the reason a modern CPU can do 16 float multiplications in one instruction. If Phase 1–5 taught you to think in scalars, this phase re-teaches your brain to think in lanes. Every serious inference kernel in 2026 — llama.cpp’s Q4_K matmul, cuBLAS’s tile scheduler, oneDNN’s convolution — is written as vector code first and scalar code never. This file gives you the vocabulary and the four practical rules.

The instruction-set landscape you actually meet¶

ISA	Width	Where you meet it	The register file
SSE / SSE2	128-bit	Every x86 since 2003; baseline for `x86_64`	`xmm0`–`xmm15`
AVX / AVX2	256-bit	Intel Haswell+ (2013), AMD Zen+	`ymm0`–`ymm15`
AVX-512	512-bit	Intel server + AMD Zen 4/5	`zmm0`–`zmm31`
AVX-VNNI	256-bit int8 dot	Intel Alder Lake+, big win for INT8 inference	shares ymm
AMX	tile	Sapphire Rapids+, matmul-specific	tile registers
NEON	128-bit	ARMv8, all Apple Silicon	`v0`–`v31`
SVE / SVE2	scalable	ARMv9 servers (Graviton3+, Grace, Nvidia Bluefield)	vector-length agnostic
SME / SME2	matrix	ARMv9-A extension, appearing 2025+	matrix tile

On your MacBook you get NEON and (if it’s Apple Silicon M-series) the additional Apple AMX private extension — not the same as Intel’s AMX, only reachable via the Accelerate framework, not intrinsics. On a Zoho Linux VM you probably get AVX2, sometimes AVX-512. llama.cpp’s build system probes for all of these at CMake time; grep CMakeLists.txt for GGML_AVX2, GGML_AVX512, GGML_NEON to see the flags.

Three ways to actually emit vector instructions¶

1. Auto-vectorization. Write clean scalar loops and hope gcc -O3 -march=native figures it out. It sometimes does. It fails whenever there’s a potential aliasing conflict, a data-dependent branch inside the loop, or a reduction the compiler can’t prove associative. Diagnose with gcc -fopt-info-vec-missed or clang -Rpass-missed=loop-vectorize. This is your baseline; anything you write by hand must beat it or you shouldn’t have written it.

2. Compiler intrinsics. Include <immintrin.h> (x86) or <arm_neon.h> (ARM) and call functions like _mm256_fmadd_ps or vfmaq_f32. The compiler treats each intrinsic as one instruction but still handles register allocation and scheduling. This is what llama.cpp uses. Look at ggml/src/ggml-cpu/ggml-cpu-quants.c — the ggml_vec_dot_q4_0_q8_0 function is a masterclass. On AVX2 it uses _mm256_maddubs_epi16, _mm256_madd_epi16, and a horizontal add to produce a scalar dot product from two blocks in ~10 instructions.

3. Hand-written assembly. Reserved for people writing OpenBLAS/BLIS micro-kernels, cuBLAS internals, or the innermost tiles of llama.cpp for a specific chip. Not your job in M10–M11. Read it, don’t write it.

The alignment gotcha¶

Aligned loads (_mm256_load_ps) require the pointer be a multiple of 32 bytes on AVX2. Unaligned loads (_mm256_loadu_ps) don’t, but on old micro-architectures they were 2× slower. On Skylake and later the penalty is near zero; on Haswell it’s real. The safe pattern:

Allocate with posix_memalign(&p, 64, size) — 64-byte alignment covers cache line, AVX-512, and future-proofs.
Or aligned_alloc(64, size) (C11).
Or _mm_malloc(size, 64) if you already have Intel intrinsics headers.
Fail loudly: assert(((uintptr_t)ptr & 63) == 0).

ggml pads every tensor to 32-byte boundaries in ggml_new_tensor_impl. Study that function; it’s how professionals do it.

The four rules of writing SIMD kernels that actually win¶

Vectorize the innermost loop only. Everything outside should be loop-nest orchestration.
Amortize loads. A vector load costs the same as a scalar load. Fetch once, multiply against 4–16 values held in registers.
Never diverge inside the vector. No if branches in the inner loop. Use masks (_mm256_blendv_ps, vbslq_f32) or predicated instructions.
Fuse multiply-add. FMA does a = a + b*c in one instruction with one rounding. Use _mm256_fmadd_ps on x86 and vfmaq_f32 on NEON. Free 2× throughput on modern chips.

Concrete llama.cpp speedups (measured, not theoretical)¶

Kernel	Scalar C	Intrinsic	Speedup	Source
Q4_0 · Q8_0 dot, block of 32	~200 cycles	~25 cycles AVX2	8×	`ggml-cpu-quants.c`
Q4_K matmul, 4096×4096	baseline	+AVX-VNNI INT8	~1.7×	llama.cpp PR discussions 2024–2025
Q8_0 quantize row of 4096 fp32	scalar	NEON on M1 Max	~5×	measured with `llama-bench`

When you write the SIMD kernel in projects.md, your target is 4× over -O3 scalar. That is the number a professional would recognize as “you’re not embarrassing yourself.” Above 8× on a single-thread GEMM tile puts you in the same neighborhood as OpenBLAS.

Debugging vector code¶

gdb’s p $ymm0.v8_float prints an AVX register as 8 floats.
lldb on Apple Silicon: register read v0 -f float32 for a NEON v register.
Use perf stat -e fp_arith_inst_retired.256b_packed_single on Linux to prove your loop is emitting 256-bit FMAs.
On Mac use Instruments’ “Metal System Trace” or sudo powermetrics for coarse counters; Apple hides most PMCs.

What most people get wrong about SIMD¶

They think SIMD means “use __m256 types.” It doesn’t. SIMD means restructuring your data layout so the vector unit can eat it. If your tensor is stored as an array of struct-of-4-floats (like a Vec4), you’re already dead — you’ll spend all your time shuffling. Store as struct-of-array (SoA), contiguous, aligned, padded. This is why ggml’s block_q4_0 is 32 nibbles packed into 16 bytes plus one FP16 scale — exactly one SIMD register worth of nibbles per block. The data format is the SIMD design.

Return to README.md · Next: 03_cache_and_memory_hierarchy.md