Phase 1: Classical Machine Learning (Months 3–4)¶
Phase 0 gave you the mathematical language. Phase 1 is where you learn to speak it fluently about actual learning algorithms. This phase is not about collecting sklearn API calls — it is about understanding why each algorithm works, when it breaks, and what the math from Phase 0 is doing inside the black box. By the end of Month 4, you will be able to derive, implement, evaluate, and deploy classical ML models — and you will have the vocabulary to read research papers that cite them.
This is the phase most practitioners either rush through (using sklearn as a magic box) or skip entirely (jumping to deep learning). Both are mistakes. The classical ML landscape is where the foundational intuitions live: bias-variance tradeoff, overfitting, feature engineering, model evaluation rigor. Neural networks are just more expressive versions of these ideas. If you don’t understand logistic regression deeply, you don’t understand a softmax classifier. If you don’t understand gradient boosting, you don’t understand residual connections.
What Phase 1 Builds On (Phase 0 Connections)¶
Phase 0 Concept |
Phase 1 Application |
|---|---|
Linear algebra (matrix multiply, SVD) |
PCA, linear regression normal equation, kernel methods |
Eigendecomposition |
PCA, covariance analysis |
Gradients, chain rule |
Gradient boosting, logistic regression optimization |
Probability, MLE |
All probabilistic models — Naive Bayes, logistic regression, GMMs |
KL divergence, cross-entropy |
Classification loss functions, model comparison |
Information gain (entropy) |
Decision tree splitting criterion |
If Phase 0 is the toolbox, Phase 1 is the first time you use every tool on a real job.
The Classical ML Landscape¶
8-Week Schedule¶
Month 3 (Weeks 1–4): Core Algorithms¶
Week |
Topic |
Hours |
Primary Resource |
|---|---|---|---|
Week 1 |
Linear + Logistic Regression (derivations) |
12–15 |
Andrew Ng Week 1–3 + Géron Ch 4 |
Week 2 |
Decision Trees + Random Forests |
12–15 |
Géron Ch 6–7 + sklearn MOOC Module 3 |
Week 3 |
Gradient Boosting (XGBoost/LightGBM) |
10–12 |
StatQuest GBM series + XGBoost paper |
Week 4 |
SVMs + Unsupervised (K-Means, PCA revisit) |
10–12 |
Géron Ch 5 + sklearn MOOC |
Month 4 (Weeks 5–8): Engineering & Projects¶
Week |
Topic |
Hours |
Primary Resource |
|---|---|---|---|
Week 5 |
Model Evaluation — rigorous protocol |
8–10 |
ESL Ch 7 (bias-variance) + sklearn docs |
Week 6 |
Feature Engineering (the real work) |
10–12 |
Kaggle Learn + Feature Engineering for ML (Alice Zheng) |
Week 7 |
Production pipeline — FastAPI, serialization |
8–10 |
FastAPI docs + sklearn Pipelines |
Week 8 |
Phase Projects (all 3) |
12–15 |
Your own implementation |
Total: ~90–110 hours over 8 weeks @ 10–15 hrs/week
Primary Resources¶
Courses¶
Resource |
Why |
Cost |
Hours |
|---|---|---|---|
Andrew Ng ML Specialization (Coursera) |
245+ Reddit thread mentions; strongest mathematical framing of classical ML available in course form; 4.9/5 stars, 120K+ reviews |
Free audit |
~33 hrs |
sklearn MOOC (mooc.datascience.paris) |
574 Reddit upvotes; “pure gold”; built by sklearn core team; free; perfect for production-grade sklearn thinking |
Free |
~25 hrs |
StatQuest YouTube (Josh Starmer) |
Community bridge resource; fill gaps when Ng’s lectures don’t stick; ~20 hrs ML playlist |
Free |
As needed |
Books¶
Resource |
Why |
Cost |
|---|---|---|
Hands-On ML (Géron, 3rd ed.) |
Community top choice for practical sklearn; covers algorithms + deployment; 82 upvotes in rigorous/practical thread |
~$50 / free with O’Reilly |
ISL with Python (James et al.) |
Strong statistical theory for classical ML; new Python edition; free PDF |
Free |
ESL (Hastie et al.) |
PhD-level rigor; Chapter 7 (bias-variance, model selection) is essential |
Free PDF |
Papers Worth Reading¶
Paper |
Why |
Where |
|---|---|---|
XGBoost (Chen & Guestrin, 2016) |
Still the most-cited applied ML paper; understand what it actually claims |
|
Random Forests (Breiman, 2001) |
Original paper; short and readable; understand what Breiman actually proved |
Available via Scholar |
Exit Criteria — What You Can Do at End of Month 4¶
You are ready to move to Phase 2 (Deep Learning) when you can honestly check all of these:
Derive linear regression analytically (normal equation) and via gradient descent — from scratch, no lookup
Explain logistic regression as maximum likelihood estimation of a Bernoulli distribution — not just “sigmoid function”
Implement a decision tree from scratch (information gain splitting, stopping criteria) — without sklearn
Explain why random forests reduce variance but not bias — with the mathematical argument
Describe what gradient boosting is actually doing (fitting residuals, additive model) — not just “it won Kaggle”
Implement k-means from scratch and explain the E-M interpretation
Apply the correct train/val/test protocol; distinguish cross-validation from test evaluation; never test on val
Compute and interpret AUC-ROC, precision-recall, and calibration curves for a real dataset
Build a full sklearn Pipeline (imputer → encoder → scaler → model) that fits on train and transforms test correctly
Serve a trained model via FastAPI with a
/predictendpointArticulate the bias-variance tradeoff as a mathematical tension, not a metaphor
Complete at least 2 of the 3 Phase Projects with their acceptance criteria met
What Most People Get Wrong in This Phase¶
❌ Mistake 1: sklearn as a Black Box¶
RandomForestClassifier(n_estimators=100).fit(X, y) is not understanding Random Forests. You need to know what happens when n_estimators goes from 10 to 1000 (variance decreases, bias unchanged, compute increases). You need to know that max_features='sqrt' is the key decorrelation mechanism between trees. You need to be able to reproduce Figure 15.1 from ESL from memory.
Fix: Implement every major algorithm from scratch in Phase Projects before using sklearn’s version.
❌ Mistake 2: Ignoring the Probabilistic Frame¶
Most learners see “logistic regression” and think “classification algorithm.” The correct frame is: “logistic regression is a discriminative model that directly estimates P(Y=1|X) using maximum likelihood, where the log-odds is a linear function of inputs.” This reframe makes the connection to neural networks, softmax, and cross-entropy loss immediately obvious.
Fix: For every algorithm, ask: “What probability distribution is this model assuming? What is the likelihood it is maximizing?”
❌ Mistake 3: Confusing Validation with Testing¶
This kills the integrity of every experiment. The test set is touched exactly once — at the very end. If you use test set performance to make any decision (hyperparameter choice, feature selection, architecture selection), you have contaminated it. This is the p-hacking equivalent in ML.
Fix: Establish the protocol before touching any data. Write it down. Follow it mechanically.
❌ Mistake 4: Feature Engineering Neglect¶
Most courses spend 80% of time on algorithms and 5% on feature engineering. The community reality (verified by Kaggle competition analysis): 60–70% of ML performance gains come from feature engineering, not algorithm selection. The algorithm is often the last thing that matters.
Fix: Treat 04_feature_engineering.md as a primary file, not an appendix.
Files in This Phase¶
File |
Content |
Priority |
|---|---|---|
|
Linear/Logistic/Trees/Forests/Boosting/SVM derivations + code |
Week 1–3 |
|
K-Means/Clustering/t-SNE/Autoencoders |
Week 4 |
|
Evaluation protocol, bias-variance, metrics |
Week 5 |
|
Encoding, missing data, scaling, selection, interactions |
Week 6 |
|
Pipelines, serialization, FastAPI, drift monitoring |
Week 7 |
|
3 concrete portfolio projects with acceptance criteria |
Week 8 |
Return to [13-Month Roadmap Root] · Next: 01_supervised_learning.md