RAG Systems: Retrieval-Augmented Generation

RAG is the most commonly deployed LLM architecture in production, and for good reason: it solves the two hardest problems with raw LLMs — knowledge cutoffs and hallucination about proprietary data — without the cost and instability of fine-tuning. The architecture is conceptually simple (retrieve relevant context, then generate) but the gap between a working prototype and a production-grade RAG system is enormous. This document covers the full stack with honest benchmarks and a production-ready code example.

---

1. Architecture Overview

A RAG pipeline has five stages:

Document Corpus
      ↓
[1. Chunking]      Split documents into retrievable units
      ↓
[2. Embedding]     Convert chunks to dense vectors via embedding model
      ↓
[3. Indexing]      Store vectors in a vector database
      ↓
[4. Retrieval]     At query time: embed query, find k nearest chunks
      ↓
[5. Generation]    Feed retrieved context + query to LLM, generate answer

Original paper: Lewis et al., 2020 — arXiv 2005.11401 (Facebook AI Research)

---

2. Chunking Strategies

Chunking is the first place most RAG systems fail silently. The granularity of your chunks directly determines retrieval precision: too small and you lose context; too large and you retrieve noise with the signal.

Fixed-Size Chunking

Split every N tokens with an M-token overlap:

from langchain.text_splitter import RecursiveCharacterTextSplitter

splitter = RecursiveCharacterTextSplitter(
    chunk_size=512,      # tokens per chunk
    chunk_overlap=64,    # overlap to preserve context at boundaries
    separators=["\n\n", "\n", ". ", " ", ""],  # split hierarchy
)
chunks = splitter.split_text(document)

When to use: Default choice. Works for 80% of cases. When it fails: Documents with heterogeneous structure (code + prose + tables).

Semantic Chunking

Split at semantic boundaries rather than fixed sizes. Use embedding similarity to detect when topic shifts:

# pip install langchain-experimental
from langchain_experimental.text_splitter import SemanticChunker
from langchain_openai import OpenAIEmbeddings

splitter = SemanticChunker(
    OpenAIEmbeddings(),
    breakpoint_threshold_type="percentile",
    breakpoint_threshold_amount=95,  # split when similarity drops to 95th percentile
)
chunks = splitter.split_text(document)

When to use: Long documents with clear topic boundaries (research papers, contracts, documentation). Cost: Requires embedding every sentence during chunking — expensive for large corpora.

Hierarchical / Parent-Child Chunking

Store large parent chunks for context, retrieve small child chunks for precision. At query time, retrieve the child chunk but return the parent for generation:

# LlamaIndex implementation
from llama_index.core.node_parser import HierarchicalNodeParser

parser = HierarchicalNodeParser.from_defaults(
    chunk_sizes=[2048, 512, 128]  # parent → intermediate → child
)

When to use: When you need precise retrieval (small chunks) but coherent generation context (large chunks). Empirically gives 10-20% better answer quality on long-document QA benchmarks.

---

3. Embedding Models (2025 State)

The embedding model is the single biggest quality lever in a RAG pipeline. A better embedding model with a worse LLM usually beats a worse embedding model with a better LLM.

Model	MTEB Score	Dim	Context	Cost	Notes
voyage-3-large	68.2	1024	32K	$0.06/1M tokens	Top overall 2025, Anthropic
text-embedding-3-large	64.6	3072	8K	$0.13/1M tokens	OpenAI, widely used
bge-m3	62.9	1024	8K	Free (open)	Best open-source, multilingual
E5-mistral-7b	66.6	4096	32K	Free (open)	High quality, high cost to run
Cohere embed-v3	64.1	1024	512	$0.10/1M tokens	Good for semantic search
nomic-embed-text	62.4	768	8K	Free (open)	Fast, reproducible, open weights

Practical recommendation for 2025:

• Production with budget: voyage-3-large (best quality-per-dollar) or text-embedding-3-large

• Open-source / on-prem: bge-m3 (multilingual) or nomic-embed-text (fast + reproducible)

• Never use text-embedding-ada-002 — it’s 3 years old and 5-10 points behind on MTEB

MTEB Leaderboard: https://huggingface.co/spaces/mteb/leaderboard — check this before committing to a model; it updates monthly.

---

4. Vector Databases: Honest Comparison

Database	Type	Latency	Scale	Cost	Best For
pgvector	Extension for PostgreSQL	~5-20ms	Up to ~10M vectors	Free + infra	Existing Postgres stack; avoids new infra

Decision framework:

• Existing Postgres: add pgvector, skip new infra

• New project, < 1M vectors: Chroma (local) → Qdrant (production)

• 1M vectors, on-prem: Qdrant

• Zero-ops tolerance: Pinecone

• Research: FAISS

---

5. Reranking

Initial vector retrieval is approximate — fast but imprecise. A reranker is a cross-encoder model that takes (query, document) pairs and computes a true relevance score. Expensive (full attention over both query and document), but run only on the top-k retrieved candidates.

Pattern: Retrieve top-50 with vector search → Rerank to top-5 with cross-encoder → Pass top-5 to LLM.

This typically improves answer quality by 10-20% on knowledge-intensive QA tasks.

from sentence_transformers import CrossEncoder

reranker = CrossEncoder("cross-encoder/ms-marco-MiniLM-L-6-v2")  # fast, good quality

def rerank(query: str, documents: list[str], top_n: int = 5) -> list[str]:
    pairs = [(query, doc) for doc in documents]
    scores = reranker.predict(pairs)
    ranked = sorted(zip(scores, documents), reverse=True)
    return [doc for _, doc in ranked[:top_n]]

Better rerankers (2025):

• BAAI/bge-reranker-v2-m3 — open-source, multilingual, strong quality

• Cohere rerank-v3 — best-in-class managed API, ~$1/1K queries

• Jina Reranker v2 — open-source, 512 token context

---

6. RAG Evaluation: RAGAS Framework

RAGAS (arXiv 2309.15217) provides automated metrics that don’t require ground-truth answers for every metric:

Metric	What it measures	Requires ground truth?
Faithfulness	Are all claims in the answer supported by the retrieved context?	No
Answer Relevance	Does the answer actually address the question?	No
Context Recall	Was all necessary information retrieved?	Yes (ground truth answer)
Context Precision	Were retrieved chunks relevant? (signal-to-noise)	Yes

# pip install ragas langchain-openai
from ragas import evaluate
from ragas.metrics import faithfulness, answer_relevancy, context_recall, context_precision
from datasets import Dataset

# Prepare evaluation dataset
eval_data = Dataset.from_dict({
    "question": ["What is PagedAttention?"],
    "answer": ["PagedAttention is a memory management technique..."],  # LLM output
    "contexts": [["PagedAttention manages KV cache in non-contiguous memory blocks..."]],  # retrieved chunks
    "ground_truth": ["PagedAttention is an attention algorithm..."],  # reference answer
})

results = evaluate(
    eval_data,
    metrics=[faithfulness, answer_relevancy, context_recall, context_precision],
)
print(results)
# Output: {'faithfulness': 0.89, 'answer_relevancy': 0.92, 'context_recall': 0.85, 'context_precision': 0.78}

Target scores for production RAG:

• Faithfulness: > 0.85 (below this = hallucination problem)

• Answer Relevance: > 0.80

• Context Precision: > 0.70 (below this = retrieval quality problem)

---

7. Advanced RAG Patterns

HyDE: Hypothetical Document Embeddings

Instead of embedding the query directly, ask the LLM to generate a hypothetical answer, then embed that. The intuition: a hypothetical answer is more similar to actual answer documents than a short question is.

def hyde_retrieve(query: str, vectorstore, llm, k: int = 5):
    hypothetical_doc = llm.invoke(
        f"Write a paragraph answering: {query}\nAnswer:"
    )
    return vectorstore.similarity_search(hypothetical_doc, k=k)

Empirically improves recall by 5-15% on knowledge-intensive tasks (Gao et al., 2022).

Self-RAG

The model decides when to retrieve and whether to use retrieved information. Adds special tokens: [Retrieve], [IsREL], [IsSUP], [IsUSE]. Requires a specially fine-tuned model. Honest assessment: high complexity, moderate gain. Worth knowing, not worth building from scratch.

Corrective RAG (CRAG)

Add a retrieval evaluator that assesses retrieved document quality. If quality is low, trigger a web search as fallback. Practical for domains with rapidly-changing information.

---

8. RAG vs. Fine-tuning: The Decision Framework

Research and practitioner consensus (2025) is clear:

Scenario	Best Approach	Why
Dynamic knowledge (news, docs updated regularly)	RAG	Can’t retrain continuously
Need citations/provenance	RAG	Retrieval provides attributable sources
Proprietary knowledge, < 200K tokens	Full-context prompting + cache	Often beats RAG complexity
Proprietary knowledge, > 200K tokens	RAG	Context window limits
Style/format/personality change	Fine-tuning (SFT)	Behavior, not knowledge
Consistent output format	Fine-tuning	More reliable than prompt engineering
Cost at scale (> 10M tokens/month)	Fine-tuning	~88% cost reduction at scale
General Q&A on public knowledge	Neither — use capable base model	Base models already know it

Key finding (2025): For knowledge bases under ~200K tokens, full-context prompting with prompt caching (Anthropic/OpenAI cache discounts) can be cheaper and faster than building RAG infrastructure. Do the math before assuming RAG is necessary.

---

9. Full RAG Pipeline Code

# pip install llama-index llama-index-embeddings-huggingface qdrant-client sentence-transformers

from llama_index.core import VectorStoreIndex, SimpleDirectoryReader, Settings
from llama_index.core.node_parser import SentenceSplitter
from llama_index.embeddings.huggingface import HuggingFaceEmbedding
from llama_index.llms.openai import OpenAI
from llama_index.core.postprocessor import SentenceTransformerRerank
from llama_index.core import QueryBundle
import os

# ── Configuration ──────────────────────────────────────────────────────
DOCS_DIR = "./documents"    # put your .pdf, .txt, .md files here
EMBED_MODEL_ID = "BAAI/bge-small-en-v1.5"  # fast, good open-source model
RERANK_MODEL_ID = "cross-encoder/ms-marco-MiniLM-L-6-v2"
LLM_MODEL = "gpt-4o-mini"   # or "claude-3-haiku" for cost optimization

# ── Global Settings ────────────────────────────────────────────────────
Settings.embed_model = HuggingFaceEmbedding(
    model_name=EMBED_MODEL_ID,
    max_length=512,
)
Settings.llm = OpenAI(model=LLM_MODEL, temperature=0.1)
Settings.chunk_size = 512
Settings.chunk_overlap = 64

# ── Load Documents ─────────────────────────────────────────────────────
documents = SimpleDirectoryReader(
    DOCS_DIR,
    required_exts=[".pdf", ".txt", ".md"],
    recursive=True,
).load_data()

print(f"Loaded {len(documents)} document(s)")

# ── Build Index ────────────────────────────────────────────────────────
# Uses in-memory FAISS by default; swap to Qdrant for production:
# from llama_index.vector_stores.qdrant import QdrantVectorStore
# import qdrant_client
# client = qdrant_client.QdrantClient(url="http://localhost:6333")
# vector_store = QdrantVectorStore(client=client, collection_name="docs")

splitter = SentenceSplitter(chunk_size=512, chunk_overlap=64)
index = VectorStoreIndex.from_documents(
    documents,
    transformations=[splitter],
    show_progress=True,
)

# ── Configure Retrieval Pipeline ───────────────────────────────────────
reranker = SentenceTransformerRerank(
    model=RERANK_MODEL_ID,
    top_n=5,  # after reranking, keep top 5
)

query_engine = index.as_query_engine(
    similarity_top_k=20,  # retrieve 20, rerank to 5
    node_postprocessors=[reranker],
    response_mode="compact",  # "tree_summarize" for long context
    verbose=True,
)

# ── Query ──────────────────────────────────────────────────────────────
response = query_engine.query(
    "What are the main findings regarding model alignment approaches?"
)

print("\n=== Answer ===")
print(response.response)
print("\n=== Source Chunks ===")
for i, node in enumerate(response.source_nodes):
    print(f"\n[{i+1}] Score: {node.score:.3f}")
    print(f"    Source: {node.metadata.get('file_name', 'unknown')}")
    print(f"    Text: {node.text[:200]}...")

# ── RAGAS Evaluation ───────────────────────────────────────────────────
# Collect ground truth pairs, then:
# from ragas import evaluate
# from ragas.metrics import faithfulness, answer_relevancy
# results = evaluate(eval_dataset, metrics=[faithfulness, answer_relevancy])
# Target: faithfulness > 0.85, answer_relevancy > 0.80

---

What Most People Get Wrong

Building RAG when they need fine-tuning, and fine-tuning when they need RAG. RAG handles knowledge gaps. Fine-tuning handles behavior gaps. If your model gives wrong answers because it doesn’t know your proprietary data, that’s a knowledge gap — use RAG. If your model gives correct information but in the wrong format, tone, or structure, that’s a behavior gap — use SFT. Most teams reach for fine-tuning first because it feels more “technical,” and end up with a model that was expensively trained to know facts that will be stale in 3 months.

The second mistake: Evaluating RAG with human-eye-scan instead of RAGAS. “It looks right” is not an evaluation protocol. Set up RAGAS on 100 test queries before deploying. Low faithfulness scores mean your model is hallucinating despite having the context. Low context precision means you’re sending noise to the LLM and wasting tokens.

---

Papers Reference

Paper	ArXiv ID	Key Contribution
Original RAG	2005.11401	RAG architecture, retrieval-augmented generation
RAGAS	2309.15217	Automated RAG evaluation framework
HyDE	2212.10496	Hypothetical document embeddings for retrieval
Self-RAG	2310.11511	Selective retrieval with learned tokens
Corrective RAG	2401.15884	Retrieval quality assessment + web fallback

---

Return to 02_parameter_efficient_finetuning.md · Next: 04_llm_agents.md