Phase 3 Projects

Three projects. Each one is designed to force a specific type of understanding that reading cannot provide. The acceptance criteria are binary — either the artifact exists and meets the bar, or it doesn’t. “I understand it conceptually” is not a passing criterion for any of these.

Estimated total time: 30–45 hours across Month 7–8, working 10–15 hrs/week alongside full-time work. Sequence them in order — each builds on the previous.

---

Project 1: nanoGPT — Character-Level Transformer from Scratch

What you’re building: A decoder-only transformer trained on Shakespeare’s complete works, generating character-by-character text. Based on Andrej Karpathy’s nanoGPT (github.com/karpathy/nanoGPT).

Why this project, not just the tutorial: Writing your own transformer forces you to confront exactly where your attention mechanism understanding breaks down. The 20 bugs you will fix in forward() are more educational than 20 hours of reading.

Specification

Dataset: Tiny Shakespeare (~1MB), available at:

wget https://raw.githubusercontent.com/karpathy/char-rnn/master/data/tinyshakespeare/input.txt

Architecture:

• Context length: 256 characters

• Embedding dim: 384

• Layers: 6

• Heads: 6 (d_head = 64)

• Feedforward dim: 4 × 384 = 1536

• Dropout: 0.2

• Total parameters: ~10M

Training:

• Optimizer: AdamW, lr=3e-4, weight_decay=0.1

• Batch size: 64

• Steps: 5000 (GPU), 2000 (CPU — expect lower final performance)

• Device: any — but record what you trained on

Acceptance Criteria (all must be met):

• Implementation is original — not copy-pasted from nanoGPT source. Reference the architecture; write the code yourself.

• Training loss curve shows clear convergence (save a plot)

• Validation perplexity < 3.5 after full training on GPU (< 5.0 on CPU acceptable)

• Model generates Shakespeare-style text that a reader could identify as “Shakespeare-like” even if grammatically imperfect

• Post a model card on HuggingFace Hub (RaghulR2003/nanogpt-shakespeare or similar) with: training loss curve, sample generated text (200+ characters), hardware used, training time

Implementation checklist (things that bite people):

• Causal mask applied correctly — future tokens must not influence current predictions (use torch.tril)

• Positional encoding added to token embeddings, not concatenated

• Weight tying: input embedding and output projection share weights (reduces parameters, improves performance)

• model.eval() and torch.no_grad() during generation

• Temperature sampling works: temperature < 1.0 makes output more deterministic; > 1.0 makes it more random

What you’ll learn from the bugs:

• The causal mask is the difference between encoder and decoder — understanding where it goes wrong makes the architecture concrete

• Numerical stability of softmax with large logits (you will likely hit NaN at some point; track it down)

• Why the √d_k scaling matters: remove it and watch your training destabilize

Common failure mode to diagnose: If perplexity plateaus above 4.5, check: (1) learning rate — try 1e-4 and 1e-3 to bracket, (2) causal mask correctness, (3) whether positional encoding is being added or concatenated.

---

Project 2: Domain-Specific Fine-Tuning (BERT or ViT)

What you’re building: A fine-tuned pretrained model that outperforms a classical ML baseline on a task from your actual work domain or personal interest. The requirement that it be domain-specific is intentional — it forces you to deal with real data preprocessing, class imbalance, and evaluation design rather than running on a clean benchmark.

Candidate tasks (pick one or define your own):

• Text classification: Zoho CRM support ticket categorization, code comment intent classification, Stack Overflow tag prediction

• Image classification: product defect detection from photos, document type classification, satellite image land-use

• NER/information extraction: extracting entities from engineering documents, parsing structured information from resumes

Required baseline: Train a classical ML model (logistic regression, SVM, or gradient boosting) on the same task using hand-crafted features (TF-IDF for text, HOG for images, etc.). Record its performance. Your fine-tuned model must beat this baseline by a statistically meaningful margin (≥2% on the primary metric).

Specification

For text tasks — use BERT:

• Model: bert-base-uncased or a domain-appropriate variant (bert-base-cased for proper nouns, roberta-base for cleaner training)

• Fine-tuning strategy: full fine-tuning (all layers) for datasets > 1000 samples; LoRA (via peft library) if compute is constrained

• Evaluation metric: match metric to task (accuracy for balanced, F1-macro for imbalanced, MCC for severely imbalanced)

For image tasks — use ViT:

• Model: google/vit-base-patch16-224 (pretrained on ImageNet-21K)

• Fine-tuning: last 4 transformer blocks + classification head minimum; full fine-tuning if >5K training samples

• Data augmentation: RandomHorizontalFlip + ColorJitter + RandomRotation(15°) — these matter for small datasets

Acceptance Criteria (all must be met):

• Classical ML baseline trained and evaluated on same train/test split (document exact split — use random_state=42)

• Fine-tuned model exceeds baseline by ≥2% on primary metric

• Confusion matrix generated and analyzed — identify which classes are hardest and why

• Model pushed to HuggingFace Hub with a model card that includes: task description, dataset description (n_train, n_test, class distribution), baseline vs. fine-tuned comparison table, inference example

• Inference works from a fresh Python environment with from transformers import pipeline

What you’ll learn from the friction:

• HuggingFace Trainer handles a lot, but custom evaluation loops reveal where the abstraction leaks

• Tokenization for text has edge cases: truncation strategy, special token placement, handling long documents (BERT max 512 tokens — what do you do with 2000-token documents?)

• For images: ViT’s ImageProcessor expects specific normalization — using raw PIL images without it will produce silently wrong results

Failure diagnosis: If fine-tuned model doesn’t beat the baseline, check in this order: (1) learning rate too high (try 2e-5), (2) train/test split is different between baseline and fine-tuned (data leakage in classical ML baseline), (3) insufficient epochs (BERT typically needs 3–5 epochs; run early stopping with patience=2), (4) class imbalance not handled.

---

Project 3: DDPM on MNIST — Generative Model from Scratch

What you’re building: A complete DDPM (Denoising Diffusion Probabilistic Model) implementation that generates recognizable MNIST digits. The code from 04_diffusion_models_foundations.md is your starting point; this project requires you to understand it well enough to debug, modify, and evaluate it.

Why this exact benchmark: MNIST is small enough to train on CPU overnight, large enough to produce meaningful results, and familiar enough that anyone can evaluate sample quality visually. This is the canonical “proof of concept” for diffusion.

Specification

Architecture:

• UNet as described in 04_diffusion_models_foundations.md

• Time embedding dimension: 256

• Channel sequence: 1 → 64 → 128 → 256 (encoder); mirror (decoder) with skip connections

Training:

• Dataset: MNIST training set (60,000 images), normalized to [-1, 1]

• Noise schedule: linear, β₁=1e-4 to β_T=0.02, T=1000

• Optimizer: AdamW, lr=2e-4

• Batch size: 128

• Epochs: 50 (GPU: ~20 minutes; CPU: ~2 hours)

Evaluation — three required artifacts:

1. Forward process visualization (forward_process.png): A single MNIST digit shown at t = 0, 100, 200, 300, 500, 700, 900, 999. Demonstrates the noising schedule works as expected.

2. Training loss curve (training_loss_curve.png): x-axis = epoch, y-axis = MSE loss. Loss must show clear monotonic decrease from ~0.08 to below 0.015. A flat loss curve means your noise or gradient flow is broken.

3. Generated samples grid (generated_samples.png): 8×8 grid of 64 generated samples. A naive observer (non-ML person) should identify ≥50% of the images as “handwritten digit-like.”

Acceptance Criteria (all must be met):

• All three artifacts generated and committed to a public GitHub repository

• Training loss converges below 0.015 (not just decreasing — must actually converge)

• Visual inspection: ≥50% of 64 generated samples identifiable as digits to a non-ML person

• README.md in the repo includes: model architecture summary, training configuration, loss curve, sample grid, training hardware and time

• Bonus (not required): implement DDIM sampling (50 steps) and show it produces comparable samples — add the comparison to the README

Debugging guide — the failure modes you will encounter:

Symptom	Root Cause	Fix
Loss NaN after a few steps	Gradient explosion in UNet	Gradient clipping: clip_grad_norm_(model.parameters(), 1.0)
Loss converges but samples are noise	Sampling loop incorrect	Check DDPM Algorithm 2 step-by-step; verify sqrt_recip_alphas indexing
All samples identical	Collapsed stochastic term	Verify z ~ N(0,I) is added at each step t > 1
Blurry smears, not digits	Loss too low (overfit to mean)	This paradox is real — try adding EMA of model weights
Loss plateaus at 0.025+	Learning rate too high	Try lr=5e-5
Forward process visualization incorrect	alphabar not decreasing to near 0	Check cumulative product: print alphabar[-1] — should be < 0.001

What you’ll learn that the paper doesn’t state clearly: The UNet’s time embedding injection is the mechanism that makes the network a conditional denoiser rather than a general denoiser. If you remove the time embedding and train, the model will try to denoise images at all noise levels simultaneously with the same parameters — it will fail because t=1 (lightly noised) and t=999 (pure noise) require completely different operations. The time embedding is how you turn one network into 1000 specialized denoisers that share weights.

---

Delivery Summary

When all three projects are complete, you will have:

Artifact	Location	Demonstrates
nanoGPT repo + model card	HuggingFace Hub	Transformer internals understood at implementation level
Domain fine-tune + model card	HuggingFace Hub	Transfer learning pipeline in production context
DDPM repo + visuals	GitHub	Generative model math → working implementation

These three artifacts together constitute a stronger signal of Phase 3 mastery than any written exam. They are the portfolio you bring to a PhD application interview, a senior ML role conversation, or a research collaboration. They also happen to be exactly the kind of evidence that distinguishes someone who read papers from someone who shipped working implementations.

---

Return to [README.md] · Previous: [05_graph_neural_networks.md]