Rung 3: Deep Learning From Scratch

Month 6 | Phase 2: Deep Learning | ~30–40 hours total

This rung exists because model.fit() is an abstraction, not understanding. The goal is to build a working neural network — capable of actual learning on real data — without touching PyTorch, TensorFlow, JAX, or any other autograd framework. Every gradient must be computed by hand, every weight update must be explicit, and every design decision must be justified in the documentation. When you are done, you will understand what backpropagation is computing and why — not as a formula you memorized, but as a mechanism you implemented and debugged.

Scope Options (Choose One — Commit Fully)

You have three valid paths for this rung. Choose based on what genuinely challenges you and produces the richest learning:

Path B: makemore Extension (Character-Level Language Model)

Karpathy’s makemore series builds progressively more complex character-level language models. The bigram and MLP versions are tutorials. For this rung:

  1. Implement the bigram model from scratch (no tutorial code — your own implementation)

  2. Implement the MLP character-level model: implement forward pass, manual backpropagation through all layers, and training loop

  3. Extend to a model of your choice: CNN-based or simple RNN — implementing backprop through time (BPTT) manually for the RNN case

  4. Generate coherent (obviously not GPT-quality, but structurally sensible) text samples

  5. Document every backward pass computation with mathematical justification

This path proves you understand sequence modeling and gradient flow through recurrent connections.

Path C: CNN From Scratch (Most Transferable)

Implement a convolutional neural network from scratch — including the convolution operation, pooling, and backprop through all layers:

  1. Implement: Linear, Conv2d (with actual sliding window convolution, not just an FFT trick), MaxPool2d, ReLU, BatchNorm1d, CrossEntropyLoss — each as a class with .forward() and .backward()

  2. Build a mini-LeNet and train it on CIFAR-10 or Fashion-MNIST

  3. Target: ≥ 70% test accuracy on CIFAR-10 (or ≥ 88% on Fashion-MNIST)

  4. Visualize: first-layer learned filters, training/validation loss curves, confusion matrix

This path is the most directly educational for anyone moving into computer vision.

Non-Negotiable Implementation Standards (All Paths)

Regardless of which path you choose, every implementation must meet these standards:

Mathematical transparency:

  • Every backward pass computation must have a comment that states the mathematical form: e.g., # dL/dW = X.T @ dL/dout with the chain rule written out in the comment

  • Loss function derivation must appear in MATH_NOTES.md or equivalent — specifically: why does cross-entropy + softmax have the clean gradient form p - y?

  • The “vanishing gradient” problem must be demonstrable: include an experiment where you show what happens to gradient magnitudes in early layers as network depth increases (even with a shallow 5-layer version)

Verification against autograd:

  • For at least one forward + backward pass configuration, verify your manually computed gradients against PyTorch’s autograd using torch.autograd.gradcheck or manual numerical gradient checking (finite differences). The tolerance should be ≤ 1e-5.

  • Include this verification as a test: tests/test_gradient_correctness.py

Actually trains:

  • Your implementation must actually converge. Not “loss went down a bit.” Convergent means: clear downward loss trend, final accuracy meaningfully above random chance, training curve visible in generated figure.

  • If it doesn’t train, you have a bug in your backprop. Find it. Debugging a non-converging neural network from scratch is one of the most educational exercises in this entire roadmap — document the bug you found.

Acceptance Criteria

  • git clone && pip install -r requirements.txt && python train.py --config config/default.yaml trains the model and generates loss curve + accuracy plot. No notebook required to run training.

  • pytest tests/test_gradient_correctness.py verifies gradient correctness against numerical gradient check or PyTorch autograd

  • Final test accuracy meets the threshold for your chosen path (≥92% MNIST / ≥70% CIFAR-10 / coherent text generation)

  • A MATH_NOTES.md exists containing: (a) derivation of cross-entropy + softmax gradient, (b) derivation of the backward pass for at least one layer type beyond Linear, (c) the chain rule written symbolically for your specific network architecture

  • Every layer’s forward() and backward() method has docstrings and inline comments

  • The README includes: what’s implemented (list of layers/components), what it achieves (metric + comparison to a simple baseline), how to run it, and a section called “What I learned” with ≥ 3 specific technical insights

  • Loss curves are generated and committed to figures/ for display in README

  • ≥ 10 meaningful commits in git history

Hard stop gate question: If you cannot derive the gradient of cross-entropy loss with respect to the pre-softmax logits (the dL/dz computation) from first principles on a whiteboard — the rung is not complete. This is the most important single gradient computation in modern deep learning. Know it cold.

Repository Structure

dl-from-scratch/
├── README.md
├── MATH_NOTES.md                    ← gradient derivations
├── requirements.txt                 ← numpy, matplotlib, pytest, torch (tests only)
├── src/
│   ├── __init__.py
│   ├── layers.py                    ← Layer classes with forward/backward
│   ├── activations.py               ← ReLU, Sigmoid, Softmax
│   ├── losses.py                    ← CrossEntropyLoss, MSELoss
│   ├── optimizers.py                ← SGD, (optionally Adam from Rung 1)
│   └── model.py                     ← Sequential-style model container
├── data/
│   └── loaders.py                   ← dataset loading utilities
├── config/
│   └── default.yaml                 ← hyperparameters (not hardcoded in train.py)
├── figures/                         ← generated training curves
├── scripts/
│   ├── train.py                     ← main training script
│   └── visualize_filters.py         ← (Path C only)
├── tests/
│   ├── test_gradient_correctness.py
│   └── test_forward_pass_shapes.py
└── notebooks/
    └── debugging_session.ipynb      ← optional: document your debugging process

The “Explain Every Line” Standard

This rung’s documentation standard is higher than Rung 1. The target reader is a smart person who knows calculus and linear algebra but has not implemented backprop before. After reading your code and MATH_NOTES, they should be able to implement their own version.

Concretely, this means:

  • The Linear layer backward pass must have a comment explaining why dL/dW = X.T @ delta and why the transpose is correct

  • The ReLU backward pass must have a comment explaining why it’s a binary mask

  • The BatchNorm backward pass (if implemented) must reference the original paper’s gradient derivation

  • The “why cross-entropy” explanation must appear somewhere: not just “we use cross-entropy,” but why it is the correct loss for a classification probability distribution

This standard is not bureaucratic overhead. Being able to write these comments is proof of understanding. If you can’t write the comment, you don’t fully understand the line.

Time Estimate

Task

Estimated Hours

Choosing path and reading reference implementations (not copying)

2–3

Core layer implementations (forward passes)

5–7

Backward pass implementations (expect debugging)

8–12

Gradient verification tests

2–3

Training loop + convergence verification

3–4

Visualization (curves, filters)

2–3

MATH_NOTES derivations

3–4

README + polish

2–3

Total

27–39 hours

The wide range on backward passes is real: debugging a non-converging from-scratch neural network is where the learning happens. Do not rush this phase.

What Weakens This Rung

  • Micrograd tutorial code with surface-level modifications: if your implementation closely resembles the tutorial code with a different dataset, a reviewer will see it immediately. The test is whether you can extend and modify, not replicate.

  • Toy dataset only: training on XOR proves nothing beyond “it runs.” MNIST or CIFAR-10 with measurable accuracy proves it actually works.

  • No gradient verification: asserting your gradients are correct without testing them against a numerical check or autograd is trusting yourself without evidence. Find the bugs before they find you.

  • MATH_NOTES as a glossary, not a derivation: writing “cross-entropy: measures difference between distributions” is a Wikipedia sentence. Writing out the gradient computation step by step is a derivation. These are not the same thing.

  • Skipping the debugging documentation: the debugging_session.ipynb or “What Went Wrong” section in README is optional in the sense that you won’t be penalized for not having it, but it is one of the highest-signal artifacts you can include. It proves you can diagnose and fix subtle numerical issues, which is a real skill that separates ML engineers from ML notebook executors.

  • Training that doesn’t converge: if your loss isn’t going down clearly, you have a bug. An interviewer who asks “how did your from-scratch implementation perform?” and receives “it kind of trained but not really well” will not be impressed. Fix it.

Return to README.md · Previous: 02_rung_2_classical_ml_battle.md · Next: 04_rung_4_transformer_lab.md