12 — llama.cpp / GGUF / Local Inference Mastery

The other half of your goal. vLLM/SGLang optimize a datacenter GPU for maximum throughput per dollar. llama.cpp optimizes heterogeneous consumer hardware — a laptop with a 4090 and 128GB DDR5, an M4 Max with 128GB unified memory, a mid-tower with dual 3090s — for maximum quality at whatever tokens/sec the memory bus will allow. Different problem, different aesthetic, deeply worth learning.

Repo: https://github.com/ggml-org/llama.cpp (recently renamed from ggerganov/llama.cpp — legacy URL redirects). Author: Georgi Gerganov (@ggerganov).

Why this matters for you specifically:

• Your Zoho on-prem customers may run 4×L40S or dual-3090-class hardware, not H100s.

• The r/LocalLLaMA community you want credibility with lives here.

• llama.cpp’s quantization art is 3–5 years ahead of the datacenter world for sub-4-bit regimes.

• CPU-only + partial-offload deployment is the reality for many enterprise deployments where GPUs are scarce.

---

Part 1 — The GGUF format (bit-level)

GGUF (“GGML Unified Format”) is the container format for models in the llama.cpp ecosystem. Read the spec: https://github.com/ggml-org/ggml/blob/master/docs/gguf.md

File structure (top-level)

┌──────────────────────────────────────────────────────────┐
│ Header                                                   │
│  magic: "GGUF" (4 bytes)                                 │
│  version: uint32 (currently 3)                           │
│  tensor_count: uint64                                    │
│  metadata_kv_count: uint64                               │
├──────────────────────────────────────────────────────────┤
│ Metadata KV entries (arbitrary count)                    │
│  each: key (length-prefixed string) + type + value       │
│  e.g., "general.architecture" = "llama"                  │
│        "llama.context_length" = 131072                   │
│        "llama.embedding_length" = 4096                   │
│        "tokenizer.ggml.tokens" = [array of strings]      │
├──────────────────────────────────────────────────────────┤
│ Tensor info entries (one per tensor)                     │
│  each: name + n_dims + shape + type (Q4_K, etc.) + offs  │
├──────────────────────────────────────────────────────────┤
│ (alignment padding, typically 32 bytes)                  │
├──────────────────────────────────────────────────────────┤
│ Tensor data (raw bytes, at absolute offsets)             │
└──────────────────────────────────────────────────────────┘

Design wins:

• Single file, everything in it (weights, tokenizer, arch, chat template).

• Extensible metadata — add new fields without breaking readers.

• Memory-mappable (mmap()) — no parsing hot loop; the OS handles paging.

• Little-endian, self-describing types.

Exercise: write a 100-line Python script that opens a GGUF file, parses the header + metadata + tensor info, prints the model card. This is a phase-4 rite of passage.

Q4_K_M — bit layout in exquisite detail

This is the format that quietly powers most of r/LocalLLaMA. Understand it and you understand the whole k-quants family.

Q4_K is a 4-bit K-quant. It groups weights into a super-block of 256 weights, which is subdivided into 8 sub-blocks of 32 weights.

┌─────────────────────────── Super-block (256 weights) ─────────────────────────┐
│                                                                              │
│  d      (fp16,   2 bytes)  ← super-block scale (of scales)                   │
│  dmin   (fp16,   2 bytes)  ← super-block min-scale                           │
│  scales (12 bytes)         ← 8 × (6-bit scale + 6-bit min), packed           │
│  qs     (128 bytes)        ← 256 × 4-bit weights = 128 bytes of quants       │
│                                                                              │
└──────────────────────────────────────────────────────────────────────────────┘
Total:                                          144 bytes per 256 weights
                                                = 4.5 bits per weight  ✓

Dequantization (per sub-block of 32 weights):

for sub_block i in 0..8:
    scale_i = d    * unpack_6bit(scales, 2*i)         # fp16 * uint6 → fp16
    min_i   = dmin * unpack_6bit(scales, 2*i + 1)     # fp16 * uint6 → fp16
    for weight j in 0..32:
        q_ij = unpack_4bit(qs, i*32 + j)              # 0..15
        w_ij = scale_i * q_ij - min_i                 # dequantized fp16

Why is this clever? Naive per-sub-block fp16 scales would cost 8×2 = 16 bytes of scale metadata per 32 weights (4 bits/weight of overhead). Storing scales as 6-bit numbers within their own scale (super-block d) gives you sub-block precision for a fraction of the storage.

Q4_K_M vs Q4_K_S — the M/S suffixes come from llama-quantize’s type-per-tensor policy:

Suffix	Meaning	Effect
_S	“small” — all Q4_K throughout	smallest file, some quality loss
_M	“medium” — Q4_K for most; Q6_K for attention.wv and half the feed_forward.w2 tensors	bigger file (~5–10%), meaningfully better quality
_L	“large” — more Q6_K, sometimes Q8_0 for embeddings	biggest of the K-M family

The M policy exists because empirical study showed wv (the value projection) and the down-projection in the MLP are the most quantization-sensitive tensors in llama-family models. Giving them 6-bit while others take 4-bit is a Pareto-optimal move.

Other k-quants at a glance

Format	Bits/weight	Super-block	Notes
Q2_K	2.5–2.6	256	aggressive; visible quality loss on small models
Q3_K_S / _M / _L	3.4–4.0	256	k-quant sweet spot for very-large models under memory pressure
Q4_K_S / _M	4.6–4.8	256	the community default for consumer GPUs
Q5_K_S / _M	5.5–5.7	256	near-lossless for most tasks
Q6_K	6.6	256	essentially lossless perceptually; used as reference or for sensitive tensors

I-quants (importance-matrix quants, IQ*)

A newer family (post-2024) using non-uniform codebooks derived from an imatrix (see below). Names: IQ1_S, IQ2_XXS, IQ2_S, IQ3_XXS, IQ4_XS, IQ4_NL.

Key differences vs K-quants:

• Non-linear codebook — the 4-bit code doesn’t decode to a uniform integer; it indexes into a small lookup table of hand-tuned values (bilinear + curved).

• Requires imatrix at quant time — unlike k-quants which work without one.

• Better quality per bit below 4 bits: IQ3_XXS (~3.06 bpw) roughly matches Q3_K_S (~3.5 bpw). At 4 bits, IQ4_XS typically beats Q4_K_S on perplexity by 0.02–0.05.

• Slower — the lookup adds latency; on CPUs with poor gather performance it can be meaningful.

When to use IQ vs K: if you’re memory-constrained and running on GPU or on a modern CPU with fast gathers, prefer IQ at ≤4 bits. If you’re on older CPUs or want maximum decode speed, prefer K.

imatrix calibration flow

The importance matrix captures per-tensor sensitivity. Workflow:

# 1. Compute imatrix using a calibration corpus
./build/bin/llama-imatrix \
  -m model-f16.gguf \
  -f calibration_corpus.txt \
  -o model.imatrix

# 2. Quantize using the imatrix
./build/bin/llama-quantize \
  --imatrix model.imatrix \
  model-f16.gguf \
  model-IQ4_XS.gguf \
  IQ4_XS

Calibration corpus discipline:

• Match the target domain. General-purpose model → use a broad corpus (WikiText, C4 subset). Code model → use code. Chat model → use conversational data.

• 20–100k tokens is usually enough. Larger doesn’t hurt but has diminishing returns.

• Avoid degenerate content. Repeated tokens, corrupted UTF-8, single-language for a multilingual target — all mislead the importance estimate.

Community-published imatrix files exist for popular calibration corpora (e.g., Bartowski’s uploads on HuggingFace) — study them.

---

Part 2 — Partial GPU offload (-ngl) mechanics

llama.cpp’s superpower for consumer hardware: you don’t need all the model on the GPU. The -ngl N flag offloads N transformer layers (from the top down) to the GPU; the rest stay on CPU.

How it works:

• Layers 0..(total_layers - N - 1): weights stay in system RAM. Forward pass runs on CPU (AVX-512/AMX/NEON kernels).

• Layers (total_layers - N)..total_layers - 1: weights uploaded to VRAM. Forward pass runs on GPU (CUDA/Metal/Vulkan/ROCm kernels).

• Activations transferred at the CPU↔GPU boundary each token.

Key subtlety — what to offload first:

1. KV cache (via --no-kv-offload by default it’s on the GPU). This is usually the biggest per-token win because attention over long contexts is memory-bandwidth-bound and GPU bandwidth ≫ DDR5.

2. Attention layers are more compute-bound and slightly less latency-sensitive than MLPs; matters less which specific transformer layers get offloaded, but by convention llama.cpp offloads the later layers (closer to output).

3. Output projection & LM head (via --override-tensor or the -ot flag) — for very large models, sometimes worth pinning to GPU explicitly.

Napkin math for offload:

Given a 70B model at Q4_K_M (≈42 GB total, ≈1 GB per layer of 80 layers) and a 24GB GPU:

• Reserve ~4 GB for KV cache + activations + overhead.

• ~20 GB available for weights = ~19 layers on GPU.

• -ngl 19 (or use llama.cpp’s auto-detection).

Expect roughly: (GPU-fraction × GPU_speed) + (CPU-fraction × CPU_speed) tok/s, with a small overhead tax from CPU↔GPU transfers each token. On this specific setup: ~4–7 tok/s is realistic — painful but usable, and dramatically better than pure CPU (~1 tok/s).

---

Part 3 — CPU inference reality

Even with -ngl, understanding pure-CPU speed matters because it’s what you fall back to and what caps partial-offload gains.

The bandwidth cap

Decode is memory-bound. On a CPU:

tokens/sec  ≈  memory_bandwidth (bytes/s)  ÷  bytes_per_token_of_weights

For an 8B model at Q4_K_M (~4.5 GB):

Platform	RAM bandwidth	Predicted tok/s
Consumer DDR5 dual-channel (~90 GB/s)	90 GB/s ÷ 4.5 GB	~20 tok/s (theoretical), ~10–15 measured
Consumer DDR4 dual-channel (~50 GB/s)	50 GB/s ÷ 4.5 GB	~11 tok/s theoretical, ~5–9 measured
Server DDR5 8-channel (~400 GB/s)	400 / 4.5	~90 theoretical, ~50–70 measured
Apple M2/M3/M4 Max (~400 GB/s unified)	400 / 4.5	~90 theoretical, ~40–60 measured

The delta between theoretical and measured is your kernel efficiency — typically 40–70% on well-optimized code.

Big-model implication: a 70B model at Q4 (~42 GB) on DDR5 dual-channel (90 GB/s) caps at ~2 tok/s. This is why the datacenter uses HBM (~3 TB/s) and this is why big models are a memory-capacity AND memory-bandwidth problem.

Instruction sets that matter for CPU decode

• AVX-512 — wide vector ops; llama.cpp exploits it for the K-quant dequant + FMA inner loops.

• AMX (Advanced Matrix Extensions) on Intel Sapphire Rapids and newer — tile matmul instructions; specific llama.cpp kernels use it. Big win on server CPUs.

• AVX2 on older machines — works fine, ~1.5–2× slower than AVX-512.

• NEON (ARM/Apple Silicon) — llama.cpp’s Apple performance is largely NEON + Accelerate framework, with Metal for GPU-side.

On ARM servers (Graviton3/4, Ampere Altra), the SVE/SVE2 kernels are increasingly good — relevant for enterprise deployments.

Prefill on CPU

CPU prefill is way better than CPU decode because prefill is compute-bound (big GEMMs, high arithmetic intensity, AMX/AVX-512 tensor units get to shine). Expect 3–10× the CPU decode rate for prefill. That means a long system prompt + short response is a workload where CPU inference is surprisingly viable.

---

Part 4 — What to actually run

llama.cpp itself

• llama-cli — REPL / single-shot generation. Great for smoke tests.

• llama-server — OpenAI-compatible HTTP server. This is the production endpoint.

• llama-bench — the honest benchmark tool. Reports prompt tok/s and gen tok/s across configs.

• llama-perplexity — the quality evaluation. Compute perplexity on WikiText-2 or your own corpus for a quant-quality bake-off.

• llama-quantize — the quantizer. Takes an F16/F32 GGUF and a target format.

• llama-imatrix — imatrix computation.

Read these source files (~1 evening each)

1. src/llama-model-loader.cpp — GGUF parsing. Corroborate the format spec against the code.

2. ggml/src/ggml-quants.c — the quantization kernels. Read quantize_row_q4_K and dequantize_row_q4_K line by line. This is where the bits meet the metal.

3. ggml/src/ggml-cuda/ — CUDA backend for GGUF. mmvq.cu (dequant + matvec) is the interesting file.

4. src/llama-context.cpp — the inference loop, KV cache, batching. Compare to vLLM’s scheduler mentally.

5. tools/server/server.cpp — the HTTP server. Simpler than vLLM’s FastAPI stack.

---

Part 5 — The quantization bake-off (Phase 4 project)

Your public write-up for r/LocalLLaMA credibility:

Setup:

• One 7–8B model (e.g., Qwen 2.5 7B Instruct, Llama 3.1 8B Instruct).

• Formats to compare: Q8_0, Q6_K, Q5_K_M, Q4_K_M, Q4_K_S, IQ4_XS, IQ3_M, Q3_K_M, Q2_K, and (bonus) an AWQ + a GPTQ + an FP8 variant.

• Hardware: whatever you have. Report it honestly.

Metrics:

1. Perplexity on WikiText-2 via llama-perplexity. Standard, reproducible.

2. KL divergence to Q8/F16 on a held-out corpus. This is r/LocalLLaMA’s favorite quality metric because perplexity misses tail distribution damage.

3. Task evals via lm-evaluation-harness — 3–5 tasks (MMLU, GSM8K, HumanEval, HellaSwag, TruthfulQA).

4. Vibe check — 20 prompts spanning creative writing, code, math, refusal. Read the outputs.

5. Speed — tokens/sec via llama-bench, batch 1 (decode) and batch-N prompt processing. Include partial-offload configs.

6. File size on disk.

Table format that gets upvotes:

Format	Size (GB)	PPL	KL to Q8	MMLU	HumanEval	tok/s (RTX 3090, all-offload)	tok/s (CPU only, DDR5)
Q8_0	8.1	6.42	0.00	68.2	62.8	84	12
Q6_K	6.6	6.44	0.02	68.0	62.5	96	15
Q4_K_M	4.9	6.51	0.09	67.6	61.2	118	22
…

Then narrate: where the knee is (usually around Q4/IQ4), which format you’d pick for which scenario, what surprised you. That prose is the actual product.

---

Part 6 — Beyond llama.cpp: the local ecosystem

Covered fully in 13_local_ecosystem.md, but at a glance:

• Ollama — the friendly wrapper. Great UX, some non-standard GGUF handling.

• LM Studio — the GUI. Downloads from HuggingFace, runs models locally. r/LocalLLaMA’s most-used front-end.

• llama-server — the pro path. OpenAI-compatible, minimal, exactly what you want if building a product on top.

• KoboldCpp / oobabooga text-generation-webui — hobbyist front-ends with more knobs.

• exllamav2/v3 — the alternative single-GPU enthusiast serving stack. Uses EXL format (different from GGUF). Faster on GPU than llama.cpp for equivalent quality; less flexible offloading. Worth benchmarking against your GGUF variants.

• MLX (Apple) — unified-memory framework for Apple Silicon. Different quant formats (MLX-quant). If you have a Mac ≥32GB, install MLX and benchmark.

---

Exit criteria for this section

1. You can write out the Q4_K_M bit layout from memory (super-block layout, scales metadata, dequant formula).

2. You’ve parsed a GGUF file in Python by hand.

3. You’ve published a quant bake-off post with real numbers.

4. You can predict, within 30%, the tokens/sec of any (model, quant, hardware) triple before running it, from memory-bandwidth arithmetic.

5. You can walk through the imatrix flow and explain why calibration corpus choice matters.

The person who has done all five is the person r/LocalLLaMA listens to, and the person Zoho’s on-prem customers can trust with their sizing decisions. That’s a distinctive profile.