Phase 2: Deep Learning Core

Months 5–6 · 10–15 hrs/week · ~60–90 hours total

The shift from classical ML to deep learning is not primarily a tooling shift — it is a shift in what you parameterize. Classical ML engineers features by hand and learns a shallow function over them. Deep learning engineers the architecture and lets the data learn the features. This distinction sounds simple, but its implications run deep: debugging, interpretability, compute requirements, and the kinds of errors you encounter are all structurally different. Phase 2 builds the foundation so firmly that no architecture in Phase 3 will feel like magic.

This phase uses PyTorch as the primary framework. The reason is not sentiment: PyTorch powers 85% of deep learning papers in top-tier venues (NeurIPS, ICML, ICLR — as of 2025), leads job postings (37.7% of ML roles vs. TensorFlow’s 32.9%), and is the native language of the entire Hugging Face ecosystem. OpenAI, Meta, Anthropic, and every major AI research lab standardizes on it. TensorFlow remains present in legacy enterprise stacks; learn to read it, but build in PyTorch.

The Core Cycle

Every training loop in deep learning — from a 2-layer MLP to GPT-4 — is exactly this cycle. The architecture changes. The loss function changes. The optimizer changes. The cycle does not.

What Most People Get Wrong

“They use frameworks without understanding backpropagation.”

The practical consequence: when your loss diverges, your gradients explode, your BatchNorm behaves differently at inference than training, or your custom loss function produces NaNs — you have no diagnostic toolkit. You will stare at hyperparameter combinations hoping something works. This is cargo-cult engineering. Phases 2 starts by building a neural network in pure numpy, before touching PyTorch. This is non-negotiable.

Module Roadmap

#

File

Core Concept

Time Estimate

01

01_neural_networks_from_scratch.md

Forward pass, loss derivation, backprop by hand

12 hrs

02

02_pytorch_fundamentals.md

Autograd, training loop, custom datasets

10 hrs

03

03_cnns_and_computer_vision.md

Convolution, ResNet, transfer learning

14 hrs

04

04_sequence_models.md

RNN, LSTM gates, BPTT, vanishing gradient

12 hrs

05

05_training_dynamics.md

LR schedules, BatchNorm, weight init

10 hrs

06

06_phase_projects.md

3 portfolio projects with acceptance criteria

ongoing

Why Not Start with PyTorch Immediately?

The question “why implement backprop by hand in numpy when PyTorch does it automatically?” misframes what you are learning. You are not learning to compute gradients faster — you are learning what gradients represent so you can debug when they vanish, explode, or point in the wrong direction. Andrej Karpathy’s micrograd project (GitHub: karpathy/micrograd, ~5,000 lines of educational value in ~150 lines of code) makes this argument concrete: autograd is not magical, it is the chain rule applied to a directed acyclic computation graph. Once you see that, every PyTorch error message becomes readable.

Key Resources

  • Karpathy’s Neural Networks: Zero to Hero — karpathy.ai/zero-to-hero.html — undisputed community #1 (2022–2025). Series covers micrograd → makemore → GPT from scratch. The nn-zero-to-hero repo has 23,000+ stars.

  • fast.ai Practical Deep Learning for Coders — course.fast.ai — top-down, project-first. Best as a complement to Karpathy, not a replacement.

  • PyTorch official documentation — pytorch.org/docs/stable/ — better than most tutorials; read it directly.

  • 3Blue1Brown Neural Networks series — visual geometric intuition for gradient descent and backprop; tens of millions of views; recommended as a visual companion.

  • Goodfellow, Bengio, Courville — “Deep Learning” (2016) — deeplearningbook.org — free online, mathematically rigorous, Chapters 6–9 map directly to this phase.

EXIT CRITERIA — Month 6 Gate

You pass Phase 2 when you can do all of the following without looking up the mechanism:

  • Derive the gradient of cross-entropy loss with respect to the pre-softmax logits

  • Explain, in terms of the computation graph, why you call optimizer.zero_grad() before loss.backward()

  • Explain what BatchNorm does differently during model.train() vs model.eval() and why this matters

  • Implement the LSTM gating equations from memory, identifying what each gate controls

  • Fine-tune a ResNet on a custom dataset and achieve >85% accuracy on CIFAR-10 from scratch

  • Generate coherent text samples from a character-level language model you trained yourself

Return to Phase 1 README · Next: 01 — Neural Networks From Scratch