LLM Alignment and Fine-Tuning

Alignment is the process of taking a capable base model and making it behave the way you actually want. The gap between “predicts next tokens” and “follows instructions helpfully and safely” is non-trivial, and the methods used to close that gap have evolved rapidly from 2022 to 2026. This document covers the four main alignment paradigms in depth — SFT, RLHF/PPO, DPO, and GRPO — with honest cost/benefit analysis, the mathematics where they matter, and runnable code.

1. Supervised Fine-Tuning (SFT)

SFT is where every alignment pipeline starts. You take a pretrained base model and train it on a dataset of (instruction, response) pairs using standard cross-entropy loss. This teaches the model the format of instruction-following before any preference optimization.

Instruction Dataset Format

The de facto standard is the chat template format, pioneered by ShareGPT and adopted by every major model family:

{
  "conversations": [
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": "Explain gradient descent in one paragraph."},
    {"role": "assistant", "content": "Gradient descent is an iterative optimization..."}
  ]
}

What makes a good SFT dataset:

  • Quality over quantity: 10K high-quality examples outperform 1M noisy ones (verified by Alpaca → Alpaca-cleaned ablations)

  • Diversity: covers the distribution of tasks you care about

  • Format consistency: mixed templates cause training instability

  • No data contamination: eval set leakage inflates benchmark scores by 3-8% (documented in multiple papers)

Key public SFT datasets (2025):

Dataset

Size

Quality

Notes

OpenHermes-2.5

1M

High

GPT-4 generated, widely used

Dolphin

3.5M

Medium-High

Uncensored, diverse

WizardLM-Evol

70K

High

Evolved instruction complexity

Alpaca-cleaned

52K

Medium

Original Alpaca + deduplication

FLAN collection

1.8K tasks

High

Multi-task, Google Research

SFT Training Loss

Standard cross-entropy on assistant tokens only (masking user/system tokens):

L_SFT = -Σ log P(y_t | y_{<t}, x)

where x is the instruction and y is the response. Masking user tokens is critical — training on those tokens causes the model to complete prompts rather than respond to them.

2. RLHF: Reinforcement Learning from Human Feedback

RLHF (Ouyang et al., 2022 — arXiv 2203.02155) is what turned GPT-3 into InstructGPT and established the modern alignment paradigm. It operates in two stages: reward model training and PPO fine-tuning.

Stage 1: Reward Model Training

Collect human preference data: show annotators pairs of model outputs (y_w, y_l) for the same prompt x, and have them choose the preferred response. Train a reward model R_φ using the Bradley-Terry model:

P(y_w ≻ y_l | x) = σ(R_φ(x, y_w) - R_φ(x, y_l))

The loss function:

L_RM = -E[(x, y_w, y_l) ~ D] [log σ(R_φ(x, y_w) - R_φ(x, y_l))]

The reward model is typically the SFT model with the final layer replaced by a scalar head. This is important: the reward model needs to understand language at the level of the policy model.

Stage 2: PPO Fine-Tuning

Optimize the policy π_θ to maximize expected reward while staying close to the SFT reference policy π_ref (via KL penalty):

max_{π_θ} E_{x~D, y~π_θ(·|x)} [R_φ(x, y) - β · KL(π_θ(·|x) || π_ref(·|x))]

The KL term is critical. Without it, the policy “reward hacks” — finds degenerate outputs that score high on R_φ but are nonsensical to humans (e.g., repetitive token sequences that exploit reward model weaknesses).

PPO memory requirement: 4 models simultaneously:

  1. Policy model (being trained)

  2. Reference model (frozen SFT model for KL computation)

  3. Value model (PPO critic, typically same size as policy)

  4. Reward model

For a 7B model at BF16: ~4 × 14GB = ~56GB minimum, before optimizer states. In practice, 80GB+ for stable training.

3. DPO: Direct Preference Optimization

DPO (Rafailov et al., NeurIPS 2023 — arXiv 2305.18290) is the mathematical insight that changes everything. Instead of training a separate reward model and running PPO, DPO directly optimizes the policy on preference data by exploiting a closed-form relationship between the optimal policy and the reward function.

The Reparameterization Trick

Start from the RLHF objective. The optimal policy under the KL-constrained reward maximization has a closed form:

π*(y|x) = (1/Z(x)) · π_ref(y|x) · exp(r*(x,y)/β)

where Z(x) is the partition function. Rearranging to solve for r*:

r*(x,y) = β · log(π*(y|x) / π_ref(y|x)) + β · log Z(x)

The key insight: the log Z(x) term cancels when you substitute this into the Bradley-Terry preference probability:

P(y_w ≻ y_l | x) = σ(β · log(π*(y_w|x)/π_ref(y_w|x)) - β · log(π*(y_l|x)/π_ref(y_l|x)))

This gives you the DPO loss — directly parameterized by the policy you’re training, no reward model needed:

L_DPO = -E[(x, y_w, y_l)] [log σ(β(log π_θ(y_w|x)/π_ref(y_w|x) - log π_θ(y_l|x)/π_ref(y_l|x)))]

Memory requirement: 2 models (policy + frozen reference). For 7B at BF16: ~28GB + optimizer states. Runs on 2×A100 40GB.

Known Limitations of DPO

  1. Length bias: DPO-trained models systematically produce longer responses (documented in multiple ablations). The model learns that length correlates with higher human preference scores. Mitigation: length-normalize the reward signal, or use SimPO (arXiv 2405.14734) which adds an explicit length penalty.

  2. Bradley-Terry assumption: DPO inherits the BT model’s assumption that preferences are transitive and can be represented by a scalar reward. IPO (Identity Preference Optimization, arXiv 2310.12036) relaxes this.

  3. Distribution sensitivity: DPO is sensitive to the quality of rejected responses. If your rejected samples are too easy to distinguish, the model learns a superficial pattern.

4. Constitutional AI / RLAIF

Constitutional AI (Bai et al., Anthropic 2022 — arXiv 2212.08073) replaces human preference labelers with an AI critic operating according to a written “constitution” — a list of principles. The model critiques its own outputs, revises them, and generates synthetic preference pairs for DPO/RLHF.

Why it matters: Scales alignment without proportional human annotation cost. Anthropic uses this at production scale. The “RLAIF” generalization (replacing human raters with AI raters) is now standard at labs that can’t afford human annotation at scale.

Practical implication for you: When building domain-specific aligned models, you can use Claude/GPT-4 as the constitutional critic to generate preference pairs cheaply. This is a legitimate production pattern, not a hack.

5. GRPO: Group Relative Policy Optimization

GRPO (DeepSeekMath — arXiv 2402.03300; extended in DeepSeek-R1 — arXiv 2501.12948) is the most significant alignment development of 2024-2025. It proved that pure RL with verifiable rewards can elicit sophisticated reasoning capabilities without any human preference data.

How GRPO Works

For each prompt x, sample G completions {y_1, …, y_G} from the current policy. Compute a group-relative advantage for each:

A_i = (r_i - mean(r_1..r_G)) / std(r_1..r_G)

The GRPO objective:

L_GRPO = -E[Σ_i min(r_t(θ) · A_i, clip(r_t(θ), 1-ε, 1+ε) · A_i)] + β · KL(π_θ || π_ref)

where r_t(θ) = π_θ(y_i|x) / π_old(y_i|x) is the probability ratio (same as PPO clipping).

The key difference from PPO: No value network (critic). The advantage is computed from the group statistics, not from a learned value function. This removes the need for a separate critic model and simplifies training significantly.

When GRPO works: Only when you have a verifiable reward signal — math problems with ground-truth answers, code that either passes tests or doesn’t, logic puzzles with binary correct/incorrect. This is why DeepSeek applied it to math and code. It doesn’t work for open-ended generation where correctness is ambiguous.

Honest Cost/Benefit Table

Method

GPU Memory (7B)

Training Complexity

Quality Ceiling

Best For

SFT only

~16GB (BF16)

Low

Medium

Format/style adaptation, task-specific fine-tune

DPO

~28GB + optimizer

Medium

High

General alignment, chat models, 2024-2026 default

PPO/RLHF

~56GB+

Very High

Highest*

Production alignment at scale, OpenAI-style

GRPO

~28GB + optimizer

Medium

High (reasoning)

Math, code, verifiable reasoning tasks

Constitutional AI

Same as DPO

Medium

High

Scalable alignment, reduces human annotation

*PPO edges DPO by ~2.5% on math and ~1.2% general benchmarks when data quality is held fixed (ICML 2024). The gap is real but often not worth the operational complexity.

Code: SFT + DPO with TRL

The TRL (Transformer Reinforcement Learning) library from HuggingFace is the practical standard.

# requirements: pip install trl transformers datasets peft accelerate bitsandbytes

from datasets import load_dataset
from transformers import AutoTokenizer, AutoModelForCausalLM, BitsAndBytesConfig
from trl import SFTTrainer, SFTConfig, DPOTrainer, DPOConfig
from peft import LoraConfig, get_peft_model
import torch

# ────────────────────────────────────────────────────────
# STAGE 1: Supervised Fine-Tuning
# ────────────────────────────────────────────────────────

MODEL_ID = "meta-llama/Llama-3.2-1B"  # Use 1B for testing, 7B for real work

# 4-bit quantization config (QLoRA setup)
bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16,
    bnb_4bit_use_double_quant=True,
)

tokenizer = AutoTokenizer.from_pretrained(MODEL_ID)
tokenizer.pad_token = tokenizer.eos_token
tokenizer.padding_side = "right"

model = AutoModelForCausalLM.from_pretrained(
    MODEL_ID,
    quantization_config=bnb_config,
    device_map="auto",
    attn_implementation="flash_attention_2",  # remove if not available
)

# LoRA config - target attention projection matrices
lora_config = LoraConfig(
    r=16,                    # rank: 8-64, higher = more capacity
    lora_alpha=32,           # scaling factor, usually 2x rank
    target_modules=["q_proj", "v_proj", "k_proj", "o_proj"],
    lora_dropout=0.05,
    bias="none",
    task_type="CAUSAL_LM",
)

# Load instruction dataset in ChatML format
# Dataset should have a "messages" column with list of {"role": ..., "content": ...}
dataset = load_dataset("HuggingFaceH4/ultrachat_200k", split="train_sft[:5000]")

sft_config = SFTConfig(
    output_dir="./sft_output",
    max_seq_length=2048,
    num_train_epochs=1,
    per_device_train_batch_size=4,
    gradient_accumulation_steps=4,  # effective batch = 16
    learning_rate=2e-4,
    lr_scheduler_type="cosine",
    warmup_ratio=0.03,
    logging_steps=10,
    save_steps=100,
    bf16=True,
    gradient_checkpointing=True,
    dataset_text_field="prompt",  # or use dataset_kwargs
    packing=False,  # set True to pack short sequences for efficiency
)

sft_trainer = SFTTrainer(
    model=model,
    train_dataset=dataset,
    peft_config=lora_config,
    processing_class=tokenizer,
    args=sft_config,
)

sft_trainer.train()
sft_trainer.save_model("./sft_output/final")

# ────────────────────────────────────────────────────────
# STAGE 2: DPO Fine-Tuning
# ────────────────────────────────────────────────────────
# DPO dataset requires: {"prompt": str, "chosen": str, "rejected": str}

dpo_dataset = load_dataset("HuggingFaceH4/ultrafeedback_binarized", split="train_prefs[:2000]")

# Use SFT model as both policy (trainable) and reference (frozen)
dpo_config = DPOConfig(
    output_dir="./dpo_output",
    num_train_epochs=1,
    per_device_train_batch_size=2,
    gradient_accumulation_steps=8,
    learning_rate=5e-5,
    beta=0.1,           # KL penalty coefficient; 0.1-0.5 typical range
    loss_type="sigmoid", # standard DPO; alternatives: "ipo", "kto_pair"
    max_length=1024,
    max_prompt_length=512,
    bf16=True,
    logging_steps=10,
)

# Load SFT-tuned model for DPO
from peft import PeftModel
from transformers import AutoModelForCausalLM

base_model = AutoModelForCausalLM.from_pretrained(MODEL_ID, quantization_config=bnb_config, device_map="auto")
policy_model = PeftModel.from_pretrained(base_model, "./sft_output/final")

dpo_trainer = DPOTrainer(
    model=policy_model,
    ref_model=None,  # None = use model with disabled adapters as reference
    train_dataset=dpo_dataset,
    processing_class=tokenizer,
    args=dpo_config,
)

dpo_trainer.train()
dpo_trainer.save_model("./dpo_output/final")

print("DPO training complete. Merge LoRA weights before inference:")
print("  merged = policy_model.merge_and_unload()")

What to monitor during DPO training:

  • rewards/chosen should increase, rewards/rejected should decrease

  • rewards/margins (chosen - rejected) should be positive and growing

  • logps/chosen and logps/rejected diverging = learning is happening

  • If kl divergence grows too fast, increase beta

Papers Reference

Paper

ArXiv ID

Key Contribution

InstructGPT / RLHF

2203.02155

First production RLHF pipeline

DPO

2305.18290

Reward-free alignment, closed-form policy optimization

Constitutional AI

2212.08073

RLAIF, scalable alignment without human raters

DeepSeekMath / GRPO

2402.03300

Group relative policy optimization

DeepSeek-R1

2501.12948

Pure RL elicits reasoning at o1-level without SFT data

KTO

2402.01306

Non-paired preference optimization

SimPO

2405.14734

DPO length bias fix

IPO

2310.12036

Beyond Bradley-Terry assumptions

What Most People Get Wrong

Treating alignment methods as interchangeable plug-ins. DPO is not “easier RLHF” that you always use instead. GRPO is not “better DPO.” They solve different problems. DPO optimizes for human preferences on open-ended text. GRPO optimizes for verifiable correctness on tasks with ground truth. If you try GRPO on a customer support chatbot where “correct” is undefined, you’ll get garbage. Match the method to the reward signal you actually have.

Return to README.md · Next: 02_parameter_efficient_finetuning.md