15 — The Fine-Tuning Stack: LoRA, QLoRA, DPO, GRPO

Papers to keep next to your desk:

  • LoRA — Hu et al., 2021 — arxiv:2106.09685

  • QLoRA — Dettmers et al., 2023 — arxiv:2305.14314

  • DPO — Rafailov et al., 2023 — arxiv:2305.18290

  • InstructGPT (RLHF) — Ouyang et al., 2022 — arxiv:2203.02155

  • GRPO / DeepSeekMath — Shao et al., 2024 — arxiv:2402.03300

  • DeepSeek-R1 — DeepSeek-AI, 2025 — arxiv:2501.12948

  • Kahneman-Tversky Optimization (KTO) — Ethayarajh et al., 2024 — arxiv:2402.01306

  • ORPO — Hong et al., 2024 — arxiv:2403.07691

Mental model. Pretraining teaches the model to predict text. Fine-tuning teaches it behavior. There are two behavioral axes: what to output (supervised fine-tuning, SFT — from labeled pairs) and what to prefer (preference optimization — from human/AI preference or verifiable reward). Modern stacks compose them: SFT gives you a passable instruction-follower; preference optimization sharpens it into something people (or reward functions) actually like. Everything below is machinery for one of these two.

Part 1 — The parameter budget question

The first fork in the road is: how many parameters do you actually update? This dominates memory, cost, and portability more than any other choice.

Method

Trainable params

Memory (7B, bf16 base)

Portability

Quality ceiling

Full FT

100% (7B)

~112 GB (bf16 grads + AdamW states)

New checkpoint per task

Highest

LoRA

0.1–1% (~10–70M)

~18 GB

Tiny adapter (~50 MB)

~95–99% of full FT for most tasks

QLoRA

Same as LoRA

~7 GB (base in NF4)

Same tiny adapter

~98% of LoRA, fits 65B on 48 GB

DoRA

Similar to LoRA + magnitude vec

~19 GB

Same adapter shape

Slightly > LoRA at same rank

Full FT + FSDP2 CPU offload

100%

~28 GB active

Full checkpoint

Highest

Prompt/Prefix tuning

<0.01%

~14 GB

Tiny embedding

Weak; niche only

Rules of thumb:

  • If you have <100k SFT examples and a single task, LoRA/QLoRA is almost always right. The quality gap to full FT is usually smaller than the noise in your eval.

  • If you’re doing multi-task or continual-training on millions of tokens of new distribution, full FT wins.

  • If you’re fine-tuning >30B on <2 nodes, QLoRA is your only reasonable option.

  • For preference optimization (DPO/GRPO/RLHF), LoRA is standard; you’re editing behavior, not rewriting the model.

Part 2 — LoRA, mechanically

The idea. Freeze the pretrained weight W_0 ∈ R^{d×k}. For each targeted linear layer, learn a low-rank update:

W = W_0 + ΔW ,   ΔW = B · A ,   A ∈ R^{r×k}, B ∈ R^{d×r}

with rank r ≪ min(d, k) (typically 8, 16, 32, 64). At init, A ~ Kaiming, B = 0, so ΔW = 0 (identity behavior). Only A and B train. At inference, you can either (a) apply B·A on the fly (small compute penalty) or (b) fuse it back: W ← W_0 + B·A (zero inference overhead).

Knobs and how to set them:

Knob

Common values

How to think about it

r (rank)

8, 16, 32, 64, 128

Higher = more capacity, more memory. Start at 16 for SFT, 32–64 for preference.

alpha

2r or 32

Scales the update: ΔW = (alpha/r) · B·A. Fixing alpha and sweeping r keeps effective LR stable.

dropout

0.05–0.1

Regularizes the adapter. Usually leave at 0.05.

target_modules

[q_proj, k_proj, v_proj, o_proj] (attention only) or + [gate_proj, up_proj, down_proj] (attention + MLP)

Attention-only saves ~half the params; MLP-included is what QLoRA paper recommended and what PEFT defaults to now. Include MLP if you can afford it — DeepSeek and community empirics show it matters.

bias

"none" usually

LoRA on biases is rarely useful.

Learning rate

1e-4 to 3e-4

~10× higher than full-FT LR because you’re updating so few params.

A minimal PEFT + Transformers setup:

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import LoraConfig, get_peft_model
import torch

model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-7B", torch_dtype=torch.bfloat16, device_map="auto")

peft_cfg = LoraConfig(
    r=16, lora_alpha=32, lora_dropout=0.05, bias="none",
    target_modules=["q_proj","k_proj","v_proj","o_proj","gate_proj","up_proj","down_proj"],
    task_type="CAUSAL_LM",
)
model = get_peft_model(model, peft_cfg)
model.print_trainable_parameters()   # 0.5–1% of total

What LoRA is not good at. Adding new capabilities the base model lacks (new language, dramatically different domain) — the low-rank subspace is too small. If your eval loss plateaus above the base and you’re sure the data is fine, try full FT before blaming the data.

Part 3 — QLoRA (NF4 + paged optimizers)

QLoRA = “run LoRA on top of a 4-bit-quantized frozen base model.” The three technical ingredients:

  1. NF4 (NormalFloat-4). A non-uniform 4-bit format whose 16 quantization levels are placed at the quantiles of a standard normal distribution — because that’s what pretrained weight distributions look like after normalization. Better fidelity than uniform int4 at zero cost. Blockwise (block size 64 or 128) with per-block scales in fp16.

  2. Double quantization. The per-block scales themselves get quantized to 8-bit with a second-level scale. Saves ~0.4 bits/param — small, but free.

  3. Paged optimizers. AdamW states (m, v — 2× fp32/param, 8 bytes) can spike GPU memory when a batch happens to have a long sequence. bnb pages optimizer states between GPU and CPU via unified memory when pressure spikes. Prevents OOM, ~5–10% slower on the pages.

The magic property: because the base model is frozen and quantized, only the LoRA adapters need gradients. So the training memory reads:

memory ≈ base_weights (NF4)  +  LoRA_weights (bf16)  +  LoRA_grads (bf16)  +  LoRA_optimizer_states (fp32)  +  activations

For a 7B, that’s roughly 3.5 GB + 0.15 GB + 0.15 GB + 0.6 GB + activations — fits in 12 GB. For 70B, ~40 GB + change — fits on a single 48 GB card. This is why QLoRA is the enabling technique for solo fine-tuners.

Setup (bitsandbytes + PEFT):

from transformers import BitsAndBytesConfig, AutoModelForCausalLM

bnb = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_use_double_quant=True,
    bnb_4bit_compute_dtype=torch.bfloat16,
)
model = AutoModelForCausalLM.from_pretrained("meta-llama/Meta-Llama-3-70B", quantization_config=bnb, device_map="auto")
# then wrap in LoRA as above; use optim="paged_adamw_8bit"

Gotchas:

  • QLoRA converges slightly slower per step (dequant overhead in every forward). Wall-clock the delta.

  • Don’t .merge_and_unload() a QLoRA adapter into a quantized base — quality suffers. Merge into a dequantized base or ship base + adapter separately.

  • Some ops (rotary, RMSNorm) are computed in the compute_dtype (bf16); the ADD-then-matmul back to NF4 is the actual quant boundary. Understanding this lets you debug precision issues.

Part 4 — DPO: preference optimization without a reward model

The classical RLHF pipeline is (i) SFT, (ii) reward model on preference pairs, (iii) PPO against the reward model with KL to the SFT reference. It works, but you’re maintaining four models (policy, ref, reward, value) and PPO is a rollout-heavy nightmare. DPO (Rafailov et al., 2023) collapses (ii)+(iii) into a single supervised objective on preference pairs directly.

The derivation in one line. The optimal policy for the KL-regularized RL objective has a closed-form solution in terms of the reward: π*(y|x) ∝ π_ref(y|x) · exp(r(x,y)/β). Solve for r, substitute into the Bradley-Terry preference likelihood, and the reward model disappears. What’s left is:

L_DPO = -E_{(x, y_w, y_l) ~ D}  log σ( β · ( log(π_θ(y_w|x)/π_ref(y_w|x)) - log(π_θ(y_l|x)/π_ref(y_l|x)) ) )

Where y_w is the preferred completion, y_l the rejected one, π_ref is the frozen SFT model, and β (typically 0.1–0.5) controls how far you drift from the reference.

What you need:

  • An SFT model (π_ref).

  • Preference pairs — human labels, LLM-as-judge, or synthetic. Quantity: 10k–100k for a small task, more for broad behavior.

  • Compute: DPO is O(1× SFT forward) with a small constant. You keep two model copies in memory (policy + frozen ref) — or LoRA on top of a shared frozen base, which is standard now.

TRL’s DPOTrainer is the reference. Config knobs that matter:

  • beta: 0.1 is the paper default; higher = more conservative (stays closer to ref); lower = more aggressive (better preference win rate, more risk of gibberish/hacking).

  • loss_type: "sigmoid" (original), "ipo" (Azar et al., 2023 — fixes overfitting on deterministic pairs), "kto" (single-label, no pairs needed).

  • max_length / max_prompt_length: preference datasets have long completions; blow past your original context and DPO silently mistrains.

What DPO is not. It’s not RL. There is no environment, no exploration, no on-policy sampling. If your preferences are noisy or your reference model is weak, DPO amplifies noise. It also can’t optimize verifiable rewards (math answer correct/incorrect) — for that, you want GRPO.

Part 5 — GRPO: verifiable-reward RL that finally works at scale

Group Relative Policy Optimization (DeepSeekMath, then DeepSeek-R1). GRPO is what PPO wanted to be for LLMs: no critic, no value network, just group-level advantage estimation.

The setup. For each prompt x, sample a group of G completions {y_1, ..., y_G} from the current policy. Score each with a rule-based or model-based reward (r_i). Compute group-normalized advantages:

A_i = (r_i - mean(r)) / std(r)      // normalize within the group

Then optimize a PPO-style clipped surrogate against A_i, with a KL penalty to the reference:

L = -E [ min( ratio · A , clip(ratio, 1-ε, 1+ε) · A ) ] + β · KL(π_θ || π_ref)

where ratio = π_θ(y|x) / π_θ_old(y|x). The mean-and-std normalization inside each group is the “critic-free advantage estimation” trick — you use the group as its own baseline.

Why it beats PPO for LLMs:

  • No value model → half the memory, half the moving parts.

  • No GAE horizon issues; a completion is a completion.

  • Group normalization eats reward-scale drift for free.

  • Trivially parallelizes over prompts (each group is independent).

Why it’s expensive anyway. You still need G (usually 4–16) rollouts per prompt per step. If your prompt has a long chain-of-thought, that’s G × 4096-token generations per step. This is where rollout infrastructure (next file) becomes the bottleneck: your training step now contains an inference workload, and the ratio of rollout time to weight-update time is often 10:1 or worse. Fixing that is the current hot niche in ML systems.

When GRPO shines. Tasks with a verifiable reward: math (exact-match answer), code (unit tests pass), tool use (correct API call succeeds), reasoning traces (final answer correct). This is exactly the DeepSeek-R1 recipe, and it’s exactly the shape of many enterprise problems you’d tackle at Zoho — an agentic tool call is either right or wrong; a CRM query either returns the right rows or not.

When to use what:

Situation

Choose

No preference data, just supervised outputs

SFT (LoRA/QLoRA)

Human preference pairs, ≤ 100k, offline

DPO

Preference pairs but ratings noisy

IPO or KTO

Verifiable rewards (math, code, tool use)

GRPO

Frontier scale + willing to run PPO infra

RLHF/PPO

Preference + SFT rolled into one loss

ORPO

Part 6 — Data quality > algorithmic sophistication

The blunt truth from every practitioner:

“The best DPO run on average data loses to a mediocre SFT run on great data.”

Data-quality checklist for SFT:

  • Deduplicate at prompt-level and near-duplicate (MinHash/simhash). Repeated examples silently oversample and destroy generalization.

  • Contamination check against your eval sets — n-gram overlap against MMLU/GSM8K/HumanEval, mandatory.

  • Length distribution — histogram your (prompt, completion) lengths; a long tail of 8k-token examples with max_length=2048 will silently truncate half your signal.

  • Tokenizer fit — if you’re fine-tuning Llama-3 but your training corpus is heavy Chinese, expect worse token efficiency than a Qwen base would give you. Match the tokenizer to the domain.

  • Format consistency — chat templates matter. If half your SFT data uses <|im_start|> and half uses [INST], the model learns “neither.”

For preference data (DPO/GRPO):

  • Preference agreement rate — sample 200 pairs, have two humans independently label; if agreement < 70%, your preferences are noise and DPO will amplify.

  • Reward hacking hunt — sample 100 completions from the RL-tuned model; if you see repetition, formatting exploits, or refusal-then-compliance patterns, your reward has a hole.

  • Length bias — DPO reliably makes outputs longer. If your win-rate goes up but your users complain about verbosity, that’s why. Regularize with length-controlled DPO variants or bake length into the reward.

Part 7 — Evaluation discipline

Every fine-tune is a hypothesis; every hypothesis needs a before/after measurement. The minimum bar:

  1. General-capability regression battery. Run lm-evaluation-harness on 5–10 tasks before and after. If MMLU drops 3 points, you paid for your DPO gains in world-knowledge damage.

  2. Task-specific eval. The metric you actually care about — win rate vs GPT-4 on your task, exact match on your test set, human-eval on 100 samples.

  3. Perplexity on held-out chat. Cheap, catches obvious regressions.

  4. Free-form generation samples. 20 fixed prompts, greedy decode, diff against the base and the previous iteration. Human eyes catch the failure modes evals miss (mode collapse, formatting weirdness, refusal drift).

  5. Length + refusal stats. Mean/p95 completion length before and after; refusal rate on a diverse prompt set.

  6. A/B in production shadow traffic if you have it — your Zoho harness makes this feasible in a way it isn’t for most researchers. Take advantage.

Part 8 — Practical stack recommendations

For an SFT + LoRA run on a single 24GB GPU (Qwen-7B):

  • Framework: TRL’s SFTTrainer or Axolotl.

  • Base: Qwen/Qwen2.5-7B-Instruct (or -Base if you want to control the chat template).

  • Quantization: bf16 for base if it fits (~14 GB); NF4 QLoRA if 14+activations > 24 GB.

  • LoRA: r=16, alpha=32, target all linear layers.

  • Optimizer: paged_adamw_8bit, lr=2e-4, cosine schedule, warmup 3%.

  • Batch: micro=4, grad-accum to reach effective 64–128.

  • Precision: bf16 mixed; enable gradient checkpointing if OOM.

  • Framework accel: torch.compile on the base model if PEFT+compile plays nice; otherwise skip.

For a DPO run on 2× 80GB GPUs (70B QLoRA):

  • Framework: TRL’s DPOTrainer.

  • Base: your SFT’d 70B with LoRA already applied — DPO on top of SFT-LoRA using the same adapters.

  • beta=0.1, loss_type="sigmoid" first pass, "ipo" if you see overfitting.

  • FSDP2 shard, activation checkpointing on.

  • Batch: micro=1, grad-accum 32.

  • Sample completions from base + tuned every 500 steps; look at them.

For a GRPO run on 8× H100:

  • Framework: verl (from HybridFlow) or TRL’s GRPOTrainer, or OpenRLHF. See next file.

  • Rollout via vLLM engine inside the training loop.

  • Reward: rule-based (exact match, regex, tool-call schema check) whenever possible — model-based rewards are hackable.

  • G=8 rollouts, temperature 0.7–1.0 (need exploration).

  • KL β = 0.001–0.01 (very low; the group-norm advantage is already stabilizing).

  • Watch for reward hacking after step ~500. Always.

Part 9 — What to skip

The fine-tuning literature is enormous and 80% of it is noise. Things you can safely skip until proven necessary:

  • Prompt/prefix tuning. Almost always worse than LoRA and doesn’t merge.

  • Adapters (Houlsby-style). Superseded by LoRA in every practical setting.

  • RLAIF-only pipelines. Fine for research; brittle in production.

  • Any preference method with three-plus loss terms (SLiC, RRHF, PRO, RSO). If DPO/IPO/KTO can’t do the job, you probably have a data problem, not a loss problem.

Part 10 — Zoho angle

Your CRM agentic harness is a perfect fine-tuning target. The pattern:

  1. Instrument the harness to log (prompt, completion, tool_calls, user_signal) for six weeks. user_signal = did the user accept the suggested action / edit it / reject it. This is free preference data. You already have the infrastructure to do this ethically (on-prem, respect the deployment tenant’s boundaries).

  2. SFT on the accepted completions — a 7B model fine-tuned on 20k accepted examples will beat prompted-GPT-4 on your specific CRM tasks for latency-sensitive on-prem scenarios.

  3. DPO with (accepted, rejected) pairs — this gives you the second-order polish.

  4. GRPO for the tool-call format — reward = “did the tool call succeed and return non-empty results?” — verifiable, hackable-in-limited-ways, and produces measurable win rate improvements.

Each step is publishable inside Zoho R&D. Each step, done honestly, becomes a portfolio artifact when you want to move.

Exercises

  1. Take Qwen2.5-1.5B, apply LoRA r=16 on all linear modules, and fine-tune on 5k Alpaca-style examples. Measure: (i) trainable params vs total, (ii) memory peak, (iii) MMLU delta, (iv) subjective quality on 20 held-out prompts.

  2. Repeat exercise 1 with QLoRA (NF4). Explain the memory delta before running, then verify.

  3. On the same base + LoRA, run DPO with 5k synthetic preferences (generate them by having a bigger model rank two completions of Qwen for each prompt). Measure win rate vs base on a held-out set of 200 prompts (judged by yet another model). Notice how the win rate correlates with completion length.

  4. Design a GRPO experiment for a verifiable task on 1× GPU: GSM8K math. Sketch: rollout via vLLM local server, reward = final-answer exact match. What is the wall-clock ratio between rollout time and gradient update time? What would speed up rollout by 3×?

  5. Read the DeepSeek-R1 paper and identify: (a) exactly which RL algorithm they use, (b) what the reward function is at each stage, (c) how they handle mode collapse / language mixing. Write a one-page critique.

References

Next: 16_rollout_infra.md — the training loop that contains an inference engine. This is where your two career worlds fuse.