06 — Phase 2 Projects¶

Phase 2 · Months 5–6 | Total estimated time: 35–50 hours

Theory is debt. Projects are payment. The three projects below are not exercises — they are selection criteria. A recruiter or senior engineer reviewing your GitHub can distinguish someone who read about backprop from someone who implemented it. These projects are designed to produce the second kind of evidence, and they are sequenced so each one is a prerequisite for the next phase.

Complete all three before moving to Phase 3. In order.

Project 1 — Implement micrograd¶

What You’re Building¶

Andrej Karpathy’s micrograd is a minimal scalar-valued autograd engine in ~150 lines of Python. You build the Value class that wraps a scalar, tracks its computational graph, and implements backward() using topological sort. Then you extend it, build a multi-layer perceptron on top of it, and verify it against PyTorch.

This is not a toy exercise. If you cannot build micrograd, you do not understand backpropagation — you have memorized the word “backprop” and trusted a framework to do it for you. That is a fragile foundation for everything in Phase 3 and beyond.

Starting point: https://github.com/karpathy/micrograd — read the source, then close it and rebuild from scratch.

Scope¶

Your implementation must include:

Core operations with correct gradient rules: +, *, -, /, ** (power), exp, tanh, relu
Backward pass: topological sort of the computation graph, call each node’s _backward() in reverse order
MLP class: Layer and MLP built from Neuron objects using your Value engine
Training loop: SGD on a toy 2-class dataset until convergence

Extension task (required): Add at least two operations not in the original: log, sigmoid or softmax. Derive the gradient rule by hand, implement it, verify against PyTorch.

Acceptance Criteria¶

All three must be met before you mark this project complete:

(a) Gradient verification: For a fixed computation y = f(x1, x2, ...), your Value.grad must match torch.autograd gradients to within 1e-5 on at least 10 different input combinations. Write a pytest function that asserts this automatically.
(b) MLP convergence: Train your 2-layer MLP ([4, 4, 1]) on the make_moons dataset from sklearn. Achieve >95% training accuracy within 100 epochs of full-batch gradient descent.
(c) GitHub commit: Commit to a public repository with a README.md that explains: what each gate/operation’s gradient rule is and why, why topological sort is required for backward(), and what “the chain rule applied recursively” means in code.

Portfolio Rung¶

Rung 3 of 10. This is the proof-of-foundations artifact. A senior engineer who sees this repo and a clean explanation of _backward() for multiplication will take you seriously in an interview. One who doesn’t see it will quiz you on it and you will fail.

Time Estimate¶

8–12 hours. If you spend more than 12 hours, you are overthinking it — the original source is 157 lines. Your extended version should be <300 lines total.

Project 2 — Train a CNN on CIFAR-10, Beat 85% Accuracy¶

What You’re Building¶

A convolutional neural network trained from scratch on CIFAR-10 (60,000 images, 10 classes, 32×32 RGB) that achieves >85% test accuracy using only PyTorch — no pretrained weights, no external architectures.

Why 85%? Baseline reference points:

Approach	Typical Accuracy
Random chance	10%
Naive 2-layer CNN, no tricks	~65–70%
CNN + BatchNorm	~78–80%
CNN + BatchNorm + augmentation	~82–85%
CNN + BatchNorm + augmentation + LR schedule	>85% ← target
ResNet-20 (well-tuned)	~91–92%

Hitting 85% requires you to correctly implement all three: BatchNorm, data augmentation, and a learning rate schedule. Each component’s removal should drop you below the threshold. If you hit 85% without one of them, document why — that is the more interesting outcome.

Architecture Guidance¶

You are not required to use any specific architecture, but here is a baseline that reliably crosses 85%:

Input (3×32×32)
→ Conv(3→64, 3×3, pad=1) → BN → ReLU
→ Conv(64→64, 3×3, pad=1) → BN → ReLU → MaxPool(2×2)
→ Conv(64→128, 3×3, pad=1) → BN → ReLU
→ Conv(128→128, 3×3, pad=1) → BN → ReLU → MaxPool(2×2)
→ Conv(128→256, 3×3, pad=1) → BN → ReLU
→ GlobalAvgPool → Linear(256→10)

Adjust freely. The architecture matters less than the three components above.

Acceptance Criteria¶

All four must be met:

(a) >85% test accuracy on the official CIFAR-10 test set (10,000 images). Report the exact number, not a rounded estimate.
(b) Loss curves: Plot training loss and validation accuracy vs. epoch. The plot must show that the model did not catastrophically overfit (val acc should be within ~5% of train acc at peak).
(c) Confusion matrix: Per-class accuracy showing which classes the model struggles with (cat and dog are notoriously hard on CIFAR-10; automobile and truck are often confused). Identify the two worst-performing classes and state a hypothesis for why.
(d) Ablation table: Run at least one ablation experiment and document the result:

Config	Test Accuracy
Full model (BN + augmentation + LR schedule)	XX.X%
No BatchNorm	XX.X%
No augmentation	XX.X%
No LR schedule (constant LR)	XX.X%

What Weakens This Project¶

Hitting 85% without understanding why renders the project worthless as a learning artifact and unconvincing as a portfolio piece. The ablation table is not optional — it is the evidence that you know which components did the work.

If you hit 90%+ and cannot explain the confusion matrix or the ablation results, a technical interviewer will figure that out in 3 minutes.

Time Estimate¶

15–20 hours. Budget 2–3 hours for architecture iteration, 8–10 hours for training runs (use a GPU via Google Colab, Kaggle, or your local machine), and 2–3 hours for documentation and the ablation study.

Project 3 — Character-Level Language Model¶

What You’re Building¶

An LSTM or MLP-based character-level language model that generates coherent text after training on a real corpus. This project bridges Phase 2 (sequence models) to Phase 3 (the transformer), because the character-level transformer — nanoGPT — is the natural next step after this project.

Option A (recommended): Character-level LSTM trained on the tinyshakespeare corpus (~1M characters). This is the implementation from 04_sequence_models.md — extend it, document it, and produce a clean GitHub repo.

Option B (harder, more valuable): Karpathy’s makemore progression: bigram model → MLP → LSTM. This is the full arc of language model evolution from a frequency table to a recurrent network. Reference: https://github.com/karpathy/makemore. You implement each step yourself (not copy-paste) and document what changes and why.

Option B is harder but produces a more compelling portfolio artifact — you can demonstrate understanding of why each architecture step improves perplexity, not just that it does.

Corpus Options¶

Any UTF-8 text corpus works. Choose one that is:

Large enough to learn from: >500K characters
Interesting enough to make generated samples visually compelling

Corpus	Characters	Notes
tinyshakespeare	~1M	Classic; everyone uses it; easy to compare against
WikiText-2 (first 1M chars)	~1M	More varied vocabulary
Your own domain text	Any size	E.g., Python source code generates Python-like code — impressive to show

Acceptance Criteria¶

All three must be met:

(a) Coherent generation: Generated text must contain recognizable English words, reasonable capitalization, and structural patterns (dialogue format for Shakespeare, paragraph structure for prose). Post a 200-character sample in your README. A purely random character generator produces xt#qpw! — your model must clearly not do that.
(b) Perplexity tracking: Train/val perplexity documented per epoch. Report:
- Bigram baseline perplexity (count-based bigram model, ~25–35 on Shakespeare)
- Your model’s perplexity at epoch 1 (~55–70)
- Your model’s perplexity at convergence (~12–20 for LSTM on tinyshakespeare)
- The model must beat the bigram baseline by at least 2 epochs in
(c) Clean, commented code: The core sequence mechanics — pack_padded_sequence, hidden state initialization, generation via sampling — must be explicitly commented explaining what each does and why. A reader who has completed 04_sequence_models.md should be able to follow the code without external reference.

The Bridge to Phase 3¶

When this project is working, the next natural question is: “What if instead of an LSTM hidden state, I used attention over all previous characters?” That question is nanoGPT. You have the data pipeline, the training loop, the generation code, and the perplexity baseline already. Phase 3 replaces the LSTM block with a transformer block and keeps everything else.

This is the exact progression Karpathy designed in the “Neural Networks: Zero to Hero” series. Follow it.

Time Estimate¶

12–18 hours. More time goes into getting perplexity to track cleanly and generating samples that are visually compelling — don’t underestimate documentation time.

Summary Table¶

#	Project	Core Skill Proven	Acceptance Bar	Time
1	micrograd	Backpropagation from scratch	Gradients match PyTorch to 1e-5; 95% on toy dataset	8–12h
2	CIFAR-10 CNN	Full CNN training pipeline	>85% test acc + ablation study	15–20h
3	Char LM	Sequence modeling + generation	Beats bigram baseline; coherent generated text	12–18h

Total Phase 2 project time: 35–50 hours across the 2-month phase, running in parallel with the study material. This is ~4–6 hours/week of project work on top of 4–6 hours/week of reading — consistent with the 10–15 hours/week target for this roadmap.

Finish all three. Commit all three to GitHub. Move to Phase 3.

Return to README.md · Next: ../04_phase_3_modern_architectures/README.md

06 — Phase 2 Projects¶

Project 1 — Implement micrograd¶

What You’re Building¶

Scope¶

Acceptance Criteria¶

Portfolio Rung¶

Time Estimate¶

Where to Share¶

Project 2 — Train a CNN on CIFAR-10, Beat 85% Accuracy¶

What You’re Building¶

Architecture Guidance¶

Acceptance Criteria¶

What Weakens This Project¶

Time Estimate¶

Project 3 — Character-Level Language Model¶

What You’re Building¶

Corpus Options¶

Acceptance Criteria¶

The Bridge to Phase 3¶

Time Estimate¶

Summary Table¶