Parameter-Efficient Fine-Tuning (PEFT)

Full fine-tuning a 7B model requires storing and updating ~28GB of parameters in BF16, plus optimizer states (Adam = 2× parameter memory), totaling ~84GB of GPU RAM. That’s three A100s just to fit the model. PEFT methods break this constraint by updating a tiny fraction of parameters while keeping the rest frozen — achieving 80-95% of full fine-tune quality at 1-10% of the computational cost. This is not a compromise. For most domain adaptation tasks, it’s the correct engineering choice.

---

1. LoRA: Low-Rank Adaptation

Paper: Hu et al., ICLR 2022 — arXiv 2106.09685

The Mathematical Derivation

The core insight: pre-trained model weights contain a lot of “knowledge” in a high-dimensional space, but the update required for a specific task lives in a much lower-dimensional subspace. The hypothesis is that weight updates during fine-tuning have low intrinsic rank.

For a pre-trained weight matrix W₀ ∈ ℝ^(d×k), instead of computing the full update ΔW ∈ ℝ^(d×k), LoRA constrains it to a low-rank decomposition:

ΔW = BA

where:

• B ∈ ℝ^(d×r) (d rows, r columns)

• A ∈ ℝ^(r×k) (r rows, k columns)

• r << min(d, k) — the rank, a hyperparameter

The modified forward pass:

h = W₀x + ΔWx = W₀x + BAx

Initialization: A is initialized with random Gaussian, B is initialized to zero. This ensures ΔW = 0 at the start of training — the model begins identical to the pretrained checkpoint.

Scaling: LoRA includes a scaling factor α/r:

h = W₀x + (α/r) · BAx

Setting lora_alpha = 2 × r is a common heuristic. The α/r scaling keeps the magnitude of the update independent of the rank choice.

Parameter Count

For a weight matrix W₀ ∈ ℝ^(4096×4096) (e.g., a Q or V projection in LLaMA-7B):

• Full fine-tune: 4096 × 4096 = 16.8M parameters

• LoRA with r=16: 4096×16 + 16×4096 = 131K parameters — 128× reduction

For a full LLaMA-7B model with LoRA applied to all attention projections (q, k, v, o) and both MLP projections across 32 layers at r=16:

• Trainable parameters: ~8M (0.1% of 7B total)

Why It Works

Empirical evidence from the LoRA paper and subsequent work: the “intrinsic dimensionality” of fine-tuning tasks is surprisingly small. For most NLP tasks, rank r=4 captures the essential update. r=16 is the conservative-but-safe choice. r=64+ is rarely necessary and may overfit on small datasets.

A later analysis (Aghajanyan et al., 2020 — arXiv 2012.13255) measured intrinsic dimensionality directly: MRPC needs ~200 dimensions for 90% of full fine-tune performance. Out of 125M parameters in BERT, 200 effective dimensions. LoRA exploits this empirical fact.

Which Layers to Target

Standard practice (verified by ablations):

• Attention projections (q_proj, k_proj, v_proj, o_proj): primary target, always include

• MLP projections (gate_proj, up_proj, down_proj): adding these helps on code/math tasks

• Embedding layers: rarely worth it for fine-tuning

• LM head: only if changing vocabulary

target_modules = ["q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj"]

---

2. QLoRA: 4-bit Quantization + LoRA

Paper: Dettmers et al., NeurIPS 2023 — arXiv 2305.14314

QLoRA makes 7B model fine-tuning feasible on a single consumer GPU (RTX 3090/4090, 24GB). It combines three innovations:

Innovation 1: NF4 (NormalFloat4) Quantization

Standard 4-bit quantization (INT4) distributes 16 quantization levels uniformly. But neural network weights follow approximately a normal distribution — most values cluster near zero. Uniformly-spaced bins waste capacity in the tails.

NF4 places quantization levels at the quantiles of a standard normal distribution, optimally using all 16 bins:

Quantile levels: {-1.0, -0.6962, -0.5251, -0.3949, -0.2844, -0.1848, -0.0912, 0.0,
                   0.0796, 0.1609, 0.2461, 0.3379, 0.4407, 0.5626, 0.7230, 1.0}

This is information-theoretically optimal for normally distributed data. Result: NF4 matches BF16 model quality within 0.1-0.3 perplexity points on standard benchmarks — far better than INT4.

Innovation 2: Double Quantization

The quantization constants themselves (one per block of weights) consume memory: for a 7B model with 64-element blocks, that’s 7B/64 × 32 bits = ~437MB just for constants. Double quantization quantizes the quantization constants, saving ~0.4 bits per parameter. Minor in practice but adds up.

Innovation 3: Paged Optimizers

GPU memory spikes during gradient checkpointing when processing long sequences. QLoRA uses NVIDIA’s unified memory to page optimizer states to CPU RAM during spikes, preventing OOM errors. This is transparent — you don’t configure it explicitly.

VRAM Requirements Table

Model Size	Full BF16	LoRA (BF16)	QLoRA (NF4)	QLoRA + Grad Ckpt
1B	~2GB	~2GB + opt	~0.5GB	~0.5GB
3B	~6GB	~6GB + opt	~2GB	~2GB
7B	~14GB	~14GB + opt	~5GB	~5GB
13B	~26GB	~26GB + opt	~10GB	~10GB
34B	~68GB	~68GB + opt	~20GB	~20GB
70B	~140GB	~140GB + opt	~40GB	~40GB

Optimizer states (Adam): 2× parameter memory in FP32. Gradient checkpointing recomputes activations during backward pass to trade compute for memory.

Bottom line: A single RTX 4090 (24GB) can fine-tune a 7B model with QLoRA. A single A100 80GB can fine-tune a 34B model with QLoRA.

---

3. Adapter Layers

Adapters (Houlsby et al., 2019 — arXiv 1902.00751) insert small bottleneck modules into each transformer layer. The adapter is a down-projection → nonlinearity → up-projection with a residual connection:

h = h + W_up · ReLU(W_down · h)

where W_down ∈ ℝ^(d×r) and W_up ∈ ℝ^(r×d) with r << d.

Why LoRA has largely replaced adapters:

• Adapters add inference latency (extra computation even after training)

• LoRA can be merged into the base weights: W = W₀ + BA, resulting in zero inference overhead

• LoRA typically achieves comparable quality with less overhead

Adapters remain relevant when you need to swap adaptation quickly (multi-task serving, switching between many fine-tuned variants on a single base model), as they can be swapped independently.

---

4. Prefix Tuning and Prompt Tuning

Prefix Tuning (Li & Liang, 2021 — arXiv 2101.00190) prepends trainable “virtual tokens” to the key-value cache at every layer. These aren’t real tokens — they’re directly-optimized embedding vectors that steer attention.

Prompt Tuning (Lester et al., 2021 — arXiv 2104.08691) is a simplified version: trainable embeddings prepended only to the input layer.

When they work: Large models (>10B parameters) on tasks where the base model already has the capability and only needs task direction. They fail badly on small models and fail catastrophically on tasks requiring new knowledge.

In practice: Rarely used in 2025. LoRA dominates because it works across model sizes, requires no architectural assumptions, and the merge-ability eliminates inference overhead. Include these in your knowledge base but don’t spend implementation time here.

---

5. When PEFT is Sufficient vs. When You Need Full Fine-Tuning

This is the most practically important question and the one most engineers get wrong.

Scenario	Recommendation	Reasoning
New instruction format	LoRA r=8-16	Format is low-rank; small update sufficient
Domain vocabulary/style	LoRA r=16-32	Style adaptation is low-rank
Classification on domain text	LoRA r=8	Task adaptation only
Math/reasoning improvement	GRPO + LoRA or full FT	Capability improvements may need full FT
Teaching truly new knowledge	Full fine-tune	New knowledge requires updating many parameters
Multilingual → monolingual	LoRA r=32-64	Language shift needs more capacity
New modality integration	Full fine-tune	Structural changes required
>1M training examples	Full fine-tune	At scale, full FT beats PEFT quality

The honest threshold: If your task is “make the model behave differently given information it already knows” → PEFT is sufficient. If your task is “give the model knowledge it doesn’t have” → you need full fine-tuning, and even then, RAG is often the better answer (see 03_rag_systems.md).

---

Practical: Fine-tune LLaMA-3 on 8GB GPU with QLoRA

# requirements: pip install trl transformers peft accelerate bitsandbytes datasets
# Tested on RTX 3070 8GB, LLaMA-3.2-1B (use 3B or 7B on 16GB+)

import torch
from datasets import load_dataset
from transformers import (
    AutoTokenizer,
    AutoModelForCausalLM,
    BitsAndBytesConfig,
    TrainingArguments,
)
from peft import LoraConfig
from trl import SFTTrainer, SFTConfig, DataCollatorForCompletionOnlyLM

# ── Configuration ──────────────────────────────────────────────────────
MODEL_ID = "meta-llama/Llama-3.2-1B-Instruct"  # swap to 3B if you have 16GB
DATASET_NAME = "HuggingFaceH4/ultrachat_200k"
MAX_SEQ_LENGTH = 1024
OUTPUT_DIR = "./qlora_output"

# ── 4-bit Quantization ─────────────────────────────────────────────────
bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",       # NormalFloat4 — optimal for normal distributions
    bnb_4bit_compute_dtype=torch.bfloat16,
    bnb_4bit_use_double_quant=True,  # quantize the quantization constants too
)

# ── Model and Tokenizer ────────────────────────────────────────────────
tokenizer = AutoTokenizer.from_pretrained(MODEL_ID, trust_remote_code=True)
tokenizer.pad_token = tokenizer.eos_token
tokenizer.padding_side = "right"  # required for SFT loss masking

model = AutoModelForCausalLM.from_pretrained(
    MODEL_ID,
    quantization_config=bnb_config,
    device_map={"": 0},  # single GPU
    torch_dtype=torch.bfloat16,
)
model.config.use_cache = False           # disable KV cache during training
model.config.pretraining_tp = 1          # tensor parallelism = 1 for single GPU

# ── LoRA Config ────────────────────────────────────────────────────────
peft_config = LoraConfig(
    r=16,
    lora_alpha=32,
    lora_dropout=0.05,
    bias="none",
    task_type="CAUSAL_LM",
    target_modules=[
        "q_proj", "k_proj", "v_proj", "o_proj",
        "gate_proj", "up_proj", "down_proj",  # include MLP for better performance
    ],
)

# ── Dataset ────────────────────────────────────────────────────────────
dataset = load_dataset(DATASET_NAME, split="train_sft[:2000]")  # small for demo

def format_prompt(example):
    """Apply the model's chat template."""
    return tokenizer.apply_chat_template(
        example["messages"],
        tokenize=False,
        add_generation_prompt=False,
    )

dataset = dataset.map(lambda x: {"text": format_prompt(x)})

# ── Training Arguments ─────────────────────────────────────────────────
training_args = SFTConfig(
    output_dir=OUTPUT_DIR,
    num_train_epochs=1,
    per_device_train_batch_size=1,      # 8GB constraint
    gradient_accumulation_steps=16,     # effective batch = 16
    gradient_checkpointing=True,        # trade compute for memory
    optim="paged_adamw_32bit",         # paged optimizer for spike handling
    learning_rate=2e-4,
    weight_decay=0.001,
    fp16=False,
    bf16=True,
    max_grad_norm=0.3,
    warmup_ratio=0.03,
    group_by_length=True,              # group similar lengths to minimize padding
    lr_scheduler_type="cosine",
    logging_steps=25,
    save_steps=200,
    max_seq_length=MAX_SEQ_LENGTH,
    dataset_text_field="text",
    packing=False,
    report_to="tensorboard",
)

# ── Trainer ────────────────────────────────────────────────────────────
trainer = SFTTrainer(
    model=model,
    train_dataset=dataset,
    peft_config=peft_config,
    processing_class=tokenizer,
    args=training_args,
)

trainer.train()

# ── Save and Merge ─────────────────────────────────────────────────────
# Save LoRA adapter only (~32MB for r=16)
trainer.save_model(f"{OUTPUT_DIR}/adapter")

# Optional: merge adapter into base weights for deployment (zero inference overhead)
# Requires loading base model in BF16 (not 4-bit) for merging
print("\nTo merge for deployment:")
print("  from peft import PeftModel")
print("  base = AutoModelForCausalLM.from_pretrained(MODEL_ID, torch_dtype=torch.bfloat16)")
print("  merged = PeftModel.from_pretrained(base, './qlora_output/adapter')")
print("  merged = merged.merge_and_unload()")
print("  merged.save_pretrained('./merged_model')")

# ── Expected GPU Memory Usage ──────────────────────────────────────────
# Model (4-bit NF4):        ~0.5 GB for 1B, ~3.5 GB for 7B
# LoRA adapters (BF16):     ~30 MB for 1B (r=16), ~120 MB for 7B (r=16)
# Optimizer states:         ~60 MB for LoRA params (paged to CPU)
# Activations (grad ckpt):  ~1-2 GB peak (depends on seq len and batch)
# Total for 1B on 8GB:      ~3 GB — plenty of headroom
# Total for 7B on 8GB:      ~6 GB — tight, requires batch_size=1 + grad_ckpt

Measured training speed (rough estimates, single GPU):

• 1B model, RTX 3070 8GB, batch_size=1, grad_acc=16: ~120 tokens/sec

• 7B model, RTX 4090 24GB, batch_size=1, grad_acc=8: ~90 tokens/sec

• 7B model, A100 40GB, batch_size=4, grad_acc=4: ~420 tokens/sec

---

What Most People Get Wrong

Confusing LoRA rank with quality. Engineers instinctively reach for r=64 or r=128, thinking “more capacity = better.” In most cases, r=8-16 is sufficient, and higher ranks just overfit on small datasets (< 50K examples). Run a sweep: train at r=4, r=8, r=16, r=32 on your dataset, evaluate on a held-out set. The quality difference between r=16 and r=64 is typically < 1% on most tasks. The compute difference is 4×. Do the math.

The second mistake: forgetting to disable the base model’s KV cache during training (model.config.use_cache = False). Leaving it enabled silently corrupts gradients during gradient checkpointing. This is a known footgun. It’s in the TRL documentation but frequently missed.

---

Papers Reference

Paper	ArXiv ID	Key Contribution
LoRA	2106.09685	Low-rank weight decomposition for fine-tuning
QLoRA	2305.14314	NF4 quantization + LoRA, 7B on single GPU
Adapter	1902.00751	Original adapter method
Prefix Tuning	2101.00190	Trainable prefix tokens at all layers
Prompt Tuning	2104.08691	Input-layer soft prompts only
Intrinsic Dimensionality	2012.13255	Why low-rank updates work

---

Return to 01_llm_alignment_and_finetuning.md · Next: 03_rag_systems.md