Rung 2: Classical ML Battle

Month 4 | Phase 1: Classical ML | ~30–40 hours total

This rung proves end-to-end classical ML competence and, more critically, analytical thinking about data and model behavior. A Kaggle competition is the right vehicle here because it provides three things that are impossible to fake: a fixed dataset you didn’t curate yourself, a public leaderboard that ranks your submission against hundreds of others, and a forcing function to actually ship a prediction. The signal is not the rank — it is the pipeline you built, the decisions you made, and the quality of the post-mortem that explains why you made them.

What a Strong Submission Looks Like

A strong submission is not one that achieves the highest score. It is one that demonstrates systematic, principled ML engineering. The leaderboard rank is a proxy; the pipeline is the actual signal.

Required pipeline components:

1. Exploratory Data Analysis (EDA) — minimum 45 minutes of genuine analysis

  • Distribution plots for every numeric feature (histograms + KDE)

  • Missing value heatmap and analysis: which features are missing, is missingness correlated with target?

  • Correlation matrix: identify collinear features (threshold: |r| > 0.85)

  • Target distribution: class balance analysis, baseline accuracy if you predict the majority class

  • At least 2 “insight findings” that influence your feature engineering decisions — document these explicitly

2. Feature Engineering — must include at least 4 non-trivial transformations Examples of non-trivial (at least one from each tier):

  • Numerical: log1p transform on right-skewed features, polynomial interaction terms (e.g., feature_A × feature_B), binning continuous features, rank-based normalization

  • Categorical: target encoding with k-fold to prevent leakage, frequency encoding, ordinal mapping with domain knowledge justification

  • Cross-feature: ratio features between related numerics, aggregation statistics grouped by a categorical (mean, std, min, max of numeric within group)

  • Temporal (if applicable): lag features, rolling windows, time-since-event

3. Model Selection and Comparison Train and cross-validate (5-fold StratifiedKFold) at minimum:

  • LightGBM (primary horse)

  • XGBoost (comparison)

  • CatBoost (especially if high-cardinality categoricals present)

  • One linear baseline: Logistic Regression with StandardScaler (establishes floor)

Report: mean CV score ± std for each model. Do not report test score until all decisions are final.

4. Hyperparameter Optimization Use Optuna for at least your primary model. Budget: 50–100 trials. Log the optimization curve (best score vs. trial number). Include the final Optuna study visualization in your post-mortem.

5. Ensemble / Stacking Combine at least 2 models using one of: simple average, weighted average (weights by CV score), or rank-average for classification. Report the ensemble CV vs. best individual model CV. If the ensemble doesn’t improve, explain why — this is more valuable than a result that improves silently.

6. Submission and Leaderboard Submit. Record your public leaderboard score. If it differs meaningfully (>0.5%) from your CV score, investigate and explain why in your post-mortem — this is a data leakage or distribution shift signal.

Acceptance Criteria

  • Public Kaggle notebook or GitHub repo contains the full pipeline (EDA → features → models → ensemble → submission)

  • Code is structured as .py modules, not a single 800-line notebook. The notebook can orchestrate the modules but the logic lives in importable code.

  • CV evaluation is rigorous: StratifiedKFold with n_splits=5, no data leakage from target encoding (must use KFold loop for target encoding, not fit on full train set)

  • At least 3 models trained and compared with CV scores reported in a table

  • Hyperparameter optimization completed with Optuna (≥ 50 trials on primary model)

  • A Kaggle submission exists and the public leaderboard score is recorded

  • Post-mortem written and published (spec below)

  • Repository README includes: competition link, metric used, final CV score, final public LB score, rank (or top X%), and a 3-bullet “key findings” section

Leaderboard rank target: Top 20–25% on the public leaderboard. This is achievable with a proper pipeline without exotic tricks. Top 10% requires weekend-level dedication. Top 5% requires domain expertise or a compute budget. Do not chase a rank at the expense of building the pipeline correctly — the pipeline is the portfolio artifact, the rank is the timestamp on it.

The Post-Mortem: Non-Negotiable

The post-mortem is not a summary of what you did. It is a structured analysis of what worked, what didn’t, and what you understand now that you didn’t understand at the start. This document is often more valuable to a hiring manager than the submission itself — it reveals how you think.

Minimum post-mortem structure (800–1500 words):

## Competition: [Name]
## Final Result: [CV score] / [LB score] / [Rank or top X%]

### Problem Framing
- What is being predicted? What metric? Why that metric and not accuracy/MSE?
- What is the baseline (predicting most common class)?
- What would a "good" score look like and why?

### Data Observations (3–5 numbered findings from EDA)
- Finding 1: [specific observation] → [decision it caused]
- ...

### Feature Engineering
- What I tried that worked (with CV delta for each)
- What I tried that didn't work (with CV delta and hypothesis for why)
- What I would try with more time

### Model Performance
| Model | Mean CV | Std | Notes |
|-------|---------|-----|-------|
| ...   | ...     | ... | ...   |

### What Broke / What Surprised Me
- [Specific thing that didn't go as expected and what you learned]

### What I Would Do Differently
- [Concrete, specific — not vague "spend more time on EDA"]

### One Thing I Now Understand Better Than Before
- [The single most important conceptual insight from this competition]

Where to publish: Kaggle notebook (makes it findable to the community) + your technical blog (Substack, Medium, or personal site). Link both in your GitHub README.

What the Leaderboard Rank Means (and Doesn’t Mean)

What it means:

  • You completed a real ML task and your model generalizes to held-out data

  • You are in the top X% of people who attempted the same problem

  • You can execute end-to-end under a fixed deadline

What it doesn’t mean:

  • Kaggle rank does not proxy production ML competence (no deployment, no monitoring, no drift)

  • Kaggle rank does not mean you understand business context (the problem and metric were given to you)

  • A top-5% rank on a Kaggle Playground is not equivalent to production experience

The correct framing in an interview: “I entered [competition], built a pipeline including [specific techniques], achieved top [X]%, and here’s what the post-mortem revealed about [specific insight].” The post-mortem is what makes this rung substantive — without it, you have a submission file and a number, which proves very little.

Time Estimate

Task

Estimated Hours

EDA

4–6

Feature engineering (iterate 2–3 rounds)

6–8

Model training + cross-validation

4–5

Hyperparameter optimization (Optuna)

3–4

Ensemble experiments

2–3

Code cleanup and structuring into modules

3–4

Post-mortem writing + publishing

4–5

Total

26–35 hours

This maps to roughly 3 weeks of Month 4 at the target pace, leaving the final week for polishing and publication.

What Weakens This Rung

  • Tutorial-following EDA: copying a Kaggle EDA notebook with your dataset substituted in signals copy-paste, not analysis. Genuine findings come from questions you asked of the data yourself.

  • Single model, no comparison: if you trained one model and submitted, you have a black box with a number. Comparison is how you demonstrate understanding of model behavior.

  • No post-mortem, or a thin one: “I built a pipeline and got X% accuracy” is a claim, not an analysis. The post-mortem is where the thinking is visible.

  • Accuracy as the only metric: use the competition metric, and also report precision, recall, F1, and AUC where appropriate. Accuracy on an imbalanced dataset is meaningless and reporting it as the primary metric reveals shallow evaluation understanding.

  • Data leakage from target encoding: the most common subtle mistake in Kaggle. If you fit target encoding on the full training set before cross-validation, every CV score is inflated. This will fail an interview if an interviewer asks you to walk through your validation setup.

  • Leaderboard-chasing at the expense of understanding: submitting 40 times with random hyperparameter tweaks instead of building principled pipeline components. The submission history is visible on Kaggle — frenetic random search without documented reasoning looks worse than fewer, justified experiments.

  • No business framing in the README: even on a synthetic Kaggle dataset, frame it. What problem class does this represent? What would the business consequence of false positives vs. false negatives be? Show you think about ML in context.

Return to README.md · Previous: 01_rung_1_math_from_scratch.md · Next: 03_rung_3_deep_learning_from_scratch.md