Phase 3: Modern Architectures (Months 7–8)

The transformer is not just another architecture. It is the moment the field discovered that attention over a sequence is a general-purpose computation primitive — one that scales with data and compute in ways no prior architecture matched. The shift from RNNs to transformers was not incremental improvement; it was a change of substrate. Understanding why that happened is the prerequisite for everything else in this phase.

RNNs process tokens sequentially: hidden state at step t depends on step t-1. This creates two compounding problems: (1) an O(n) dependency chain that cannot be parallelized across time, bottlenecking GPU utilization, and (2) an information bottleneck — the entire history of the sequence must be compressed into a fixed-size hidden vector. Attention removes both constraints. Every token computes a weighted sum over all other tokens simultaneously. Parallelism is native. The “receptive field” is global from layer one.

“Attention Is All You Need” (Vaswani et al., 2017, arXiv:1706.03762) demonstrated that you could build a competitive sequence-to-sequence model with only attention — no recurrence, no convolution. As of 2025, that paper has approximately 120,000 citations. The architecture it introduced is the backbone of BERT, GPT, ViT, Stable Diffusion’s UNet, and AlphaFold2.

Architecture Lineage

Month-by-Month Structure

Month 7 — Foundations

  • Week 1: Scaled dot-product attention derivation from scratch → 01_attention_and_transformers.md

  • Week 2: BERT vs. GPT paradigms, fine-tuning workflow → 02_bert_and_gpt_family.md

  • Week 3: Vision Transformers, CLIP, patch embeddings → 03_vision_transformers.md

  • Week 4: First project checkpoint — nanoGPT training run

Month 8 — Generative & Structured

  • Week 1: Diffusion model theory, DDPM derivation → 04_diffusion_models_foundations.md

  • Week 2: Graph Neural Networks, message passing → 05_graph_neural_networks.md

  • Week 3–4: Projects — DDPM on MNIST, ViT/BERT fine-tune on domain task → 06_phase_projects.md

Exit Criteria for Month 8

Complete all five before moving to Phase 4. These are binary pass/fail, not “I think I understand it”:

  • Attention from scratch: Implement scaled dot-product attention in PyTorch without referencing any code. Must produce numerically equivalent output to torch.nn.MultiheadAttention on a fixed random input. Tolerance: < 1e-5.

  • BERT vs. GPT first principles: Explain, in writing, why BERT uses a masked language model objective and GPT uses an autoregressive one — including what architectural constraint forces each choice and what downstream capability each produces. No LLM assistance. Hand-written is fine.

  • nanoGPT trained: Character-level transformer trained on the Shakespeare dataset. Perplexity < 3.5 on held-out set. Coherent (not necessarily meaningful) text generation from a cold prompt.

  • Domain fine-tune: ViT-B/16 or BERT-base fine-tuned on a task from your actual work or genuine personal interest. Must beat a classical ML baseline (Logistic Regression, SVM, or Random Forest) on the same feature set. Published model card on Hugging Face Hub.

  • DDPM on MNIST: Trained diffusion model that generates samples visually indistinguishable from real MNIST digits to a naive observer. Forward process visualization included. Code on GitHub.

What Most People Get Wrong

They learn the Hugging Face API before they understand what self-attention computes. This creates engineers who can call pipeline("text-classification") but cannot:

  • Explain why their model fails on out-of-distribution inputs

  • Diagnose attention saturation in long sequences

  • Read any paper that proposes a new positional encoding scheme

  • Modify an architecture for a novel task without breaking it

The sequence is non-negotiable: derivation first, abstraction second. If you can implement it naked in PyTorch, you own it. If you can only from transformers import it, you’re renting it and you don’t have the keys.

Prerequisites Check

Before starting this phase, verify you have:

  • Implemented backpropagation manually (Phase 2)

  • Trained a CNN on CIFAR-10 from scratch (Phase 2)

  • Comfortable with PyTorch nn.Module, DataLoader, training loops

  • Read and can explain the cross-entropy loss derivation

If any of those boxes are unchecked, go back. This phase will not make sense without them.

Key Papers

Paper

arXiv

Year

Why It Matters

Attention Is All You Need

1706.03762

2017

Defines the transformer

BERT

1810.04805

2018

Encoder-only pretraining paradigm

GPT-2

1501.09186

2019

Autoregressive scaling

GPT-3

2005.14165

2020

Few-shot emergent capabilities

An Image is Worth 16x16 Words (ViT)

2010.11929

2020

Attention for vision

CLIP

2103.00020

2021

Vision-language alignment

DDPM

2006.11239

2020

Diffusion model training objective

GCN

1609.02907

2016

Graph convolutional networks

GAT

1710.10903

2017

Graph attention networks

Return to [../03_phase_2_deep_learning_core/README.md] · Next: [01_attention_and_transformers.md]