06 · Reading llama.cpp

ggml-org/llama.cpp is the single most important C/C++ codebase in the applied-AI stack today. 119k GitHub stars, 20.2k forks, 1,038+ contributors, MIT-licensed, actively released every few days (build numbers b9500–b9800+ as of mid-2026). It is the engine underneath Ollama, LM Studio, GPT4All, and a growing share of on-device inference. This file is your guided tour — which files to open, in what order, and what to look for. Budget one calendar week for this reading exercise; do not attempt to “just skim” it.

Repo geography (top-level)

llama.cpp/
├── ggml/               # the tensor library (pure C)
│   ├── include/ggml.h  # public API
│   └── src/
│       ├── ggml.c              # the graph & op dispatch
│       ├── ggml-cpu/           # CPU kernels
│       │   ├── ggml-cpu.c
│       │   └── ggml-cpu-quants.c   # the SIMD dot products
│       ├── ggml-cuda/          # NVIDIA backend
│       ├── ggml-metal/         # Apple Silicon backend
│       ├── ggml-vulkan/        # cross-vendor GPU
│       ├── ggml-sycl/          # Intel oneAPI
│       └── ggml-common.h       # ALL block struct defs (Q4_0, Q8_0, K-quants, ...)
├── src/                # the LLaMA-family model code (C++)
│   ├── llama.cpp
│   ├── llama-model.cpp         # tensor loader, architecture dispatch
│   ├── llama-context.cpp       # KV cache, decoding state
│   ├── llama-sampling.cpp      # top-k, top-p, temperature, DRY, mirostat
│   └── llama-vocab.cpp         # tokenizer (BPE, WPM, SPM)
├── tools/              # binaries (CLIs)
│   ├── main/           # llama-cli
│   ├── server/         # llama-server (OpenAI-compatible HTTP)
│   ├── quantize/       # convert FP16 → Q4_K etc
│   └── llama-bench/    # official benchmark harness
├── examples/           # smaller integrations
├── gguf-py/            # Python side for gguf conversion
│   ├── gguf/constants.py
│   └── gguf/tensor_mapping.py
├── convert_hf_to_gguf.py       # HuggingFace → GGUF, ~5,000 LOC of arch maps
├── CMakeLists.txt
└── CONTRIBUTING.md

Everything you need lives in three places: ggml/src/, src/, and tools/. Read them in that order.

The seven files, in reading order

1. ggml/include/ggml.h

Start with the public API. Skim (don’t read every line) the enums (ggml_type, ggml_op), the ggml_tensor struct, and the graph construction functions (ggml_new_tensor_*, ggml_mul_mat, ggml_add, ggml_rope, ggml_soft_max). Goal: absorb the mental model — a computation is a static graph of tensor ops built up front, then executed by a backend.

2. ggml/src/ggml-common.h

Read top to bottom. This is the block-format registry from 05_quantization_kernels_in_c.md made real. See how block_q4_0, block_q8_0, block_q4_K are laid out with _Static_assert on their sizes. Understand QK_K = 256 and why K-quants use super-blocks.

3. ggml/src/ggml-cpu/ggml-cpu-quants.c

Open ggml_vec_dot_q4_0_q8_0. Read the three implementations — AVX2, NEON, and the scalar fallback — side by side. This is the kernel from 05, in production form. Grep the same file for ggml_vec_dot_q4_K_q8_K and for quantize_row_q4_0 (the reverse direction). Budget: one full evening.

4. ggml/src/ggml.c → function ggml_compute_forward_mul_mat

This is where a ggml_mul_mat node gets dispatched to the right kernel based on tensor type and backend. Follow the tile-scheduling loop; note how it slices work across threads. Cross-reference to your own blocked GEMM from 04_matmul_and_gemm.md. See how the pros do packing (ggml_from_float_t) and dispatch.

5. src/llama-model.cpp → llama_model_loader::load_all_data

How does a .gguf file on disk become tensors in memory? mmap by default, read() fallback. Note the sanity checks, alignment enforcement, and per-architecture tensor-name mapping. Then read llm_load_hparams and llm_load_tensors — how the model architecture (Llama, Qwen, Mistral, Phi, …) becomes a tensor graph.

6. src/llama-context.cpp → the decode loop

Specifically llama_decode_internal. This is the inference loop: build the compute graph for the current batch, execute it, update the KV cache. Notice: the graph is rebuilt on every decode call — ggml is not a JIT, it’s a graph interpreter. The KV cache is a big preallocated tensor keyed by position.

7. tools/server/server.cpp

How llama.cpp exposes an OpenAI-compatible HTTP API. Uses cpp-httplib (single-header). Read how it does continuous batching — multiple requests share one decode step. This is your bridge to 09_the_ml_serving_c_stack.md.

Build it locally (mandatory)

git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
cmake -B build -DCMAKE_BUILD_TYPE=Release   # Metal auto-enabled on Apple Silicon
cmake --build build -j

# Grab a small model to actually test
wget https://huggingface.co/Qwen/Qwen2.5-0.5B-Instruct-GGUF/resolve/main/qwen2.5-0.5b-instruct-q4_0.gguf
./build/bin/llama-cli -m qwen2.5-0.5b-instruct-q4_0.gguf -p "Explain SIMD." -n 128
./build/bin/llama-bench -m qwen2.5-0.5b-instruct-q4_0.gguf

On a MacBook (Apple Silicon) llama-bench will report tokens/sec for prompt processing (pp512) and text generation (tg128). Note these numbers; you’ll come back to them when you read the Metal kernels.

The contributing path

Read CONTRIBUTING.md. Key rules that surprise newcomers:

  • Do not add new ggml_type enum values without a discussion first — the type table is finite hardware-tuned inventory.

  • Every kernel change must pass test-backend-ops and test-quantize-fns.

  • Follow the existing indent/brace style — the codebase is spaces-only, 4-space indent, K&R braces on functions.

  • PRs adding a new model architecture must include changes to gguf-py/gguf/constants.py, gguf-py/gguf/tensor_mapping.py, convert_hf_to_gguf.py, and src/llama-model.cpp. This is the four-file dance.

Model architecture onboarding: Discussion #16770, authored by the person who added Qwen3-Next, walks the entire path. Bookmark it. It is a better tutorial than any blog post you’ll find. Key gotcha it flags: ggml_mul_mat(a, b) in GGML is equivalent to torch.matmul(b, a.T) — the operand order is reversed and one side is implicitly transposed. Everyone gets bitten by this at least once.

DeepWiki

https://deepwiki.com/ggml-org/llama.cpp is an auto-generated architectural tour with call graphs and cross-references. Use it as a supplement to reading the code, never as a substitute. The AI can and does hallucinate function signatures.

What most people get wrong about reading this codebase

They try to read llama.cpp (the file) first because that’s the repo name. Wrong: the file src/llama.cpp is 8,000+ lines of C++ orchestration, mostly tokenizer glue and CLI plumbing, and it will burn your motivation. The C is in ggml/src/, not src/. Start from the kernels and move outward. When you can explain ggml_vec_dot_q4_0_q8_0 on a whiteboard, you are ready to talk to a llama.cpp maintainer as a peer. Not before.

Return to README.md · Next: 07_kernel_and_driver_intro.md