BERT and the GPT Family

The transformer gave us an architecture. BERT and GPT gave us two answers to the question: what do you do with it before you have labeled data? The answers are structurally incompatible — not by accident, but because they reflect a fundamental choice about what the pretraining task should look like. Understanding that choice determines which family of models you reach for on any given problem.

---

1. BERT — Bidirectional Encoder Representations from Transformers

Paper: Devlin et al. (2018), arXiv:1810.04805
Architecture: Encoder-only transformer. BERT-base: 12 layers, 768 hidden, 12 heads, 110M parameters. BERT-large: 24 layers, 1024 hidden, 16 heads, 340M parameters.

The Masked Language Model Objective

BERT’s training task: take a sentence, randomly mask 15% of tokens, predict the masked tokens using the full remaining context.

Input:  "The [MASK] sat on the mat."
Target: "cat"

The crucial design decision: to predict [MASK], the model can see tokens on both sides. This forces bidirectional encoding — the representation of each token is built using full left and right context simultaneously. This is why BERT is encoder-only. An autoregressive decoder cannot do this; causal masking would prevent looking right.

The 15% masking breakdown (not random replacement — this is deliberate):

• 80% of the time: replace with [MASK]

• 10% of the time: replace with a random token

• 10% of the time: keep the original token unchanged

Why the 10%/10% mix? During fine-tuning, the model never sees [MASK] tokens (they only exist in pretraining). Training 100% on [MASK] would create a distribution mismatch. The random and unchanged tokens force the model to maintain useful representations for all token positions, not just the masked ones.

Next Sentence Prediction (NSP) — and Why It Was Dropped

BERT also trained on a binary task: given sentence A and sentence B, predict whether B actually follows A in the original document (50% positive, 50% random negative).

RoBERTa finding (Liu et al., 2019, arXiv:1907.11692): NSP hurts downstream performance when removed. Hypothesis: NSP is too easy — the model learns topic correlation rather than discourse structure. RoBERTa dropped NSP, trained longer on more data with larger batches, and improved substantially on every GLUE benchmark. If you’re initializing from a checkpoint today, use RoBERTa-base or RoBERTa-large, not vanilla BERT.

Fine-Tuning BERT

BERT’s [CLS] token (prepended to every input) accumulates a summary representation of the full sequence. For classification:

Input: [CLS] sentence tokens [SEP]
Classification head: Linear(768 → num_classes) applied to [CLS] output

Fine-tune the entire model (not just the head) for best results. For BERT-base: 3 epochs, lr=2e-5, batch_size=32 is the standard starting point from the original paper.

BERT still wins in 2025 for:

• Sentence embeddings (use sentence-transformers library)

• Token classification: NER, POS tagging

• Extractive question answering (predict start/end span)

• Semantic textual similarity

• Any task where you have full input available at inference time

---

2. GPT Family — Autoregressive Language Modeling

The GPT pretraining objective is simpler: predict the next token given all previous tokens.

Input:  "The cat sat on the"
Target: "mat"

This requires a decoder-only architecture with causal masking. The architectural constraint is what enables generation: given a prompt, you can sample the next token, append it, and repeat — producing text autoregressively.

GPT-1 (Radford et al., 2018)

12-layer decoder transformer, 117M parameters. First paper to demonstrate: pretrain on large text corpus with language modeling → fine-tune on downstream tasks. The fine-tuning paradigm worked. The model world noticed.

GPT-2 (Radford et al., 2019, arXiv:1501.09186)

1.5B parameters. Same architecture as GPT-1, scaled up. Key finding: at sufficient scale, the model can solve downstream tasks without any fine-tuning, just from the format of the prompt. This was the first clear signal of emergent few-shot capability. OpenAI initially delayed full release citing misuse concerns — a decision that aged as a PR gesture more than a technical barrier.

GPT-3 (Brown et al., 2020, arXiv:2005.14165)

175B parameters. The paper that made “prompting” a research area. Demonstrated that with enough parameters, few-shot performance (5-20 examples in the context window) on many tasks matched fine-tuned smaller models. This is the paper that shifted the field from “fine-tune for every task” to “prompt for every task.”

What GPT-3 actually showed: Language modeling at scale produces a model that implicitly learns to do many tasks because the pretraining data contains implicit demonstrations of them. The scaling law (Kaplan et al., 2020, arXiv:2001.08361) made this quantitatively predictable: loss decreases as a power law with compute, data, and parameters.

GPT-4 (OpenAI, 2023)

Architecture not published. Multimodal (vision + text input). Rumored to be a Mixture of Experts (MoE) architecture — multiple specialized sub-networks with a router selecting which expert handles each token. This would explain high capability with manageable inference cost per token.

Modern Decoder Architecture Defaults (2024)

Every current open-source LLM (LLaMA 3, Mistral, Gemma, Qwen) uses these departures from the original GPT:

Component	Original GPT	2024 Standard	Why
Positional encoding	Learned absolute	RoPE	True relative positions, long context
Normalization	Post-norm LayerNorm	Pre-norm RMSNorm	Training stability, fewer parameters
Activation	ReLU	SwiGLU	Better empirical performance
Attention	Multi-head (MHA)	Grouped Query Attention (GQA)	4-8x KV-cache memory reduction

GQA explained: Standard MHA has h query heads, h key heads, h value heads. GQA uses h query heads but only g key/value heads (g < h). At inference, the KV-cache (which stores keys and values for all previous tokens) is the memory bottleneck for long contexts. GQA reduces KV-cache size by h/g while maintaining most of MHA’s representational capacity. LLaMA 3 uses g=8 with h=32 — 4x KV-cache reduction.

---

3. BERT vs. GPT — The Structural Choice

Dimension	BERT (Encoder)	GPT (Decoder)
Attention	Bidirectional	Causal (left-to-right only)
Pretraining	Masked LM	Next-token prediction
Generation	Cannot generate	Native generation
Context	Full input available	Sequential only
Best for	Classification, NER, QA	Generation, chat, reasoning
2025 dominant use	Sentence embeddings, search	Chat, code, instruction following

This is not a quality difference — it’s a structural constraint. You cannot use BERT for text generation (it’s physically prevented by the architecture — there’s no causal mask and no autoregressive sampling mechanism). You can use GPT for classification but it’s suboptimal — you’re using a model trained to predict next tokens to instead predict a class label; embedding models trained contrastively are better for that.

---

4. Fine-tuning vs. Prompting — Honest Cost/Benefit

There is cargo-cult behavior around both approaches. Here is the actual analysis:

Approach	Compute Cost	Engineering Cost	Latency	When to Use
Zero-shot prompting	API call cost	Near-zero	API round-trip	Rapid prototyping, diverse one-off tasks
Few-shot prompting	Slightly higher input tokens	Low	API round-trip	5-20 examples fit in context, no infra
PEFT / LoRA	GPU hours (1 GPU, hours-days)	Medium	Own model serving	Consistent domain adaptation, proprietary data, cost control
Full fine-tuning	GPU hours (multi-GPU, days)	High	Own model serving	Maximum performance, strict data privacy, novel task structure
Continued pretraining	GPU days (multi-GPU)	Very high	Own model serving	Domain-specific vocabulary not in base model (medical, legal, code)

LoRA (Hu et al., 2021, arXiv:2106.09685): Freezes the base model, adds low-rank decomposition matrices ΔW = A·B (rank r, typically 4-64) to the attention projection layers. Only trains A and B — typically ~0.1-1% of original parameters. Merged at inference: no added latency. This is the standard PEFT method in 2025.

---

5. Hugging Face Transformers — When to Use What

from transformers import pipeline, AutoModel, AutoTokenizer, AutoModelForSequenceClassification

# pipeline() — use for demos, exploration, one-off tasks
classifier = pipeline("text-classification", model="distilbert-base-uncased-finetuned-sst-2-english")
result = classifier("This movie was genuinely excellent.")
# Returns: [{'label': 'POSITIVE', 'score': 0.9998}]

# AutoModel — use when you need embeddings or custom heads
tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased")
model = AutoModel.from_pretrained("bert-base-uncased")

inputs = tokenizer("Hello world", return_tensors="pt")
outputs = model(**inputs)
# outputs.last_hidden_state: (batch, seq_len, 768)
# outputs.pooler_output: (batch, 768) — [CLS] representation

# AutoModelForSequenceClassification — use for fine-tuning classification
model = AutoModelForSequenceClassification.from_pretrained("bert-base-uncased", num_labels=2)

pipeline() in production: Don’t. It hides the tokenization, batching, device placement, and half-precision details. Use it for exploration. In production, control all of those explicitly.

---

6. Code — Fine-Tune BERT on SST-2 Sentiment Classification

SST-2: Stanford Sentiment Treebank binary classification. 67K training examples. Baseline BERT-base-uncased accuracy: ~93%.

from datasets import load_dataset
from transformers import (
    AutoTokenizer,
    AutoModelForSequenceClassification,
    TrainingArguments,
    Trainer
)
import numpy as np
from sklearn.metrics import accuracy_score

# ── Load data ─────────────────────────────────────────────────────────────────
dataset = load_dataset("glue", "sst2")
tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased")

def tokenize(batch):
    return tokenizer(batch["sentence"], truncation=True, max_length=128, padding="max_length")

dataset = dataset.map(tokenize, batched=True)
dataset = dataset.rename_column("label", "labels")
dataset.set_format("torch", columns=["input_ids", "attention_mask", "labels"])

# ── Model ──────────────────────────────────────────────────────────────────────
model = AutoModelForSequenceClassification.from_pretrained("bert-base-uncased", num_labels=2)

# ── Metrics ────────────────────────────────────────────────────────────────────
def compute_metrics(eval_pred):
    logits, labels = eval_pred
    preds = np.argmax(logits, axis=-1)
    return {"accuracy": accuracy_score(labels, preds)}

# ── Training ───────────────────────────────────────────────────────────────────
training_args = TrainingArguments(
    output_dir="./bert-sst2",
    num_train_epochs=3,
    per_device_train_batch_size=32,
    per_device_eval_batch_size=64,
    learning_rate=2e-5,
    weight_decay=0.01,
    evaluation_strategy="epoch",
    save_strategy="epoch",
    load_best_model_at_end=True,
    report_to="none",
)

trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=dataset["train"],
    eval_dataset=dataset["validation"],
    compute_metrics=compute_metrics,
)

trainer.train()
# Expected: eval_accuracy ≈ 0.930-0.935 after epoch 3

Expected runtime: ~20 minutes on a single A100 / 40 minutes on an RTX 3090. On CPU: not recommended (hours).

---

What Most People Get Wrong

Using BERT for generative tasks — it physically cannot generate text. BERT produces a representation of your input; it does not produce a continuation of it. If you want generation, you need a decoder model.

Using GPT embeddings for semantic similarity — GPT-style models are trained to predict the next token, not to produce semantically meaningful embeddings. The last-token embedding of a GPT model is a reasonable representation but is dominated by next-token prediction artifacts. Use text-embedding-3-large (OpenAI), intfloat/e5-large-v2, or BAAI/bge-large-en-v1.5 for embedding tasks — these are trained with contrastive objectives specifically for similarity.

Treating HuggingFace accuracy as reproducible without pinning versions — transformers, datasets, accelerate, and tokenizers version combinations break in non-obvious ways. Pin all four in requirements.txt. Always.

---

Return to [README.md] · Next: [03_vision_transformers.md]