06 · Phase 1 Projects¶

Projects are the only thing that proves you learned something. Notes are cheap. A working, deployed model with a documented methodology is evidence that cannot be faked by memorization. Each project below is designed to demonstrate a specific cluster of Phase 1 competencies to a technical interviewer, a hiring manager, or your future self in six months. They progress in difficulty and portfolio impact: Titanic/House Prices gets you started, the Random Forest from scratch proves your depth, and the deployed churn model proves you can ship.

Complete all three. Put them on GitHub. Write a proper README for each one. These are your Phase 1 artifacts.

Project 2 · Random Forest from Scratch¶

What This Proves¶

You understand decision trees well enough to build them, and you understand bootstrap aggregation well enough to extend them into an ensemble. No sklearn internals. You can explain the algorithm to anyone. This is the most technically demanding project in Phase 1 and the most powerful interview differentiator.

Project Spec¶

Repository: ml-foundations-from-scratch (same repo as Phase 0 PCA project — add to it)

File: random_forest_scratch.py

# Implement the following classes with the signatures below.
# Do NOT import sklearn's DecisionTreeClassifier or RandomForestClassifier.
# numpy only.

class Node:
    """A node in the decision tree."""
    def __init__(self):
        self.feature_index: int = None    # which feature to split on
        self.threshold: float = None       # split threshold
        self.left: 'Node' = None           # left child
        self.right: 'Node' = None          # right child
        self.value: float = None           # leaf node class prediction


class DecisionTreeScratch:
    """
    Binary CART decision tree for classification.
    Splitting criterion: Gini impurity or information gain (selectable).
    """
    
    def __init__(self, max_depth: int = 5, min_samples_split: int = 2,
                 criterion: str = 'gini', max_features: int = None):
        pass
    
    def fit(self, X: np.ndarray, y: np.ndarray) -> 'DecisionTreeScratch':
        """Build tree recursively. Returns self."""
        pass
    
    def predict(self, X: np.ndarray) -> np.ndarray:
        """Predict class labels for X."""
        pass
    
    def _gini(self, y: np.ndarray) -> float:
        """Gini impurity: 1 - sum(p_k²)"""
        pass
    
    def _entropy(self, y: np.ndarray) -> float:
        """Entropy: -sum(p_k * log2(p_k))"""
        pass
    
    def _best_split(self, X: np.ndarray, y: np.ndarray) -> tuple:
        """
        Find the best (feature_index, threshold) pair.
        - If max_features is set, randomly sample that many features (for RF).
        - Returns (best_feature, best_threshold, best_gain)
        """
        pass
    
    def _build(self, X: np.ndarray, y: np.ndarray, depth: int) -> Node:
        """Recursive tree construction."""
        pass


class RandomForestScratch:
    """
    Random Forest classifier.
    - Bootstrap sampling of training data for each tree
    - Random feature subsampling at each split (max_features='sqrt' default)
    - Prediction via majority vote
    """
    
    def __init__(self, n_estimators: int = 100, max_depth: int = 5,
                 min_samples_split: int = 2, max_features: str = 'sqrt',
                 random_state: int = None):
        pass
    
    def fit(self, X: np.ndarray, y: np.ndarray) -> 'RandomForestScratch':
        """
        Train n_estimators trees on bootstrap samples.
        Each tree uses a different bootstrap sample and random feature subsets.
        """
        pass
    
    def predict(self, X: np.ndarray) -> np.ndarray:
        """Majority vote across all trees."""
        pass
    
    def predict_proba(self, X: np.ndarray) -> np.ndarray:
        """
        Fraction of trees voting for each class.
        Returns shape (n_samples, n_classes).
        """
        pass
    
    @property
    def feature_importances_(self) -> np.ndarray:
        """
        Mean decrease in impurity across all trees and all splits.
        Normalize to sum to 1.
        """
        pass

Test Suite (must pass all):

# tests/test_random_forest_scratch.py
import numpy as np
from sklearn.datasets import load_breast_cancer, load_iris
from sklearn.metrics import accuracy_score, roc_auc_score
from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestClassifier as SklearnRF
from random_forest_scratch import RandomForestScratch

def test_binary_classification_accuracy():
    """Your RF must achieve >= 90% accuracy on breast cancer."""
    X, y = load_breast_cancer(return_X_y=True)
    X_train, X_test, y_train, y_test = train_test_split(
        X, y, test_size=0.2, random_state=42, stratify=y
    )
    rf = RandomForestScratch(n_estimators=100, max_depth=10, random_state=42)
    rf.fit(X_train, y_train)
    acc = accuracy_score(y_test, rf.predict(X_test))
    print(f"Binary classification accuracy: {acc:.4f}")
    assert acc >= 0.90, f"Expected >= 0.90, got {acc:.4f}"


def test_within_5pct_of_sklearn():
    """Your RF AUC must be within 5% of sklearn's RF AUC."""
    X, y = load_breast_cancer(return_X_y=True)
    X_train, X_test, y_train, y_test = train_test_split(
        X, y, test_size=0.2, random_state=42, stratify=y
    )
    
    your_rf = RandomForestScratch(n_estimators=100, max_depth=10, random_state=42)
    your_rf.fit(X_train, y_train)
    your_auc = roc_auc_score(y_test, your_rf.predict_proba(X_test)[:, 1])
    
    sklearn_rf = SklearnRF(n_estimators=100, max_depth=10, random_state=42)
    sklearn_rf.fit(X_train, y_train)
    sklearn_auc = roc_auc_score(y_test, sklearn_rf.predict_proba(X_test)[:, 1])
    
    diff = abs(your_auc - sklearn_auc)
    print(f"Your AUC: {your_auc:.4f}, Sklearn AUC: {sklearn_auc:.4f}, Diff: {diff:.4f}")
    assert diff <= 0.05, f"AUC gap too large: {diff:.4f} > 0.05"


def test_feature_importances_sum_to_one():
    """Feature importances must be non-negative and sum to 1."""
    X, y = load_iris(return_X_y=True)
    rf = RandomForestScratch(n_estimators=50, max_depth=5, random_state=42)
    rf.fit(X, y)
    importances = rf.feature_importances_
    assert np.all(importances >= 0), "Importances must be non-negative"
    assert abs(importances.sum() - 1.0) < 1e-6, f"Importances sum to {importances.sum()}"


def test_n_estimators_effect():
    """More trees should not hurt (or minimally hurt) performance."""
    X, y = load_breast_cancer(return_X_y=True)
    X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.2, random_state=42)
    
    accs = {}
    for n in [10, 50, 100, 200]:
        rf = RandomForestScratch(n_estimators=n, max_depth=8, random_state=42)
        rf.fit(X_tr, y_tr)
        accs[n] = accuracy_score(y_te, rf.predict(X_te))
    
    print("Accuracy by n_estimators:", accs)
    # 200 trees should not be worse than 10 trees
    assert accs[200] >= accs[10] - 0.02, "More trees should not significantly hurt"

Documentation Required:

Docstring in each method explaining the mathematical operation
A 300-word writeup in the repo README: “What I learned implementing Random Forest from scratch that sklearn hides from you” — this is the most valuable part

Acceptance Criteria:

All 4 tests pass
DecisionTreeScratch and RandomForestScratch use only numpy (no sklearn algorithm classes)
Binary classification AUC within 5% of sklearn’s implementation
Feature importances sum to 1.0 (within 1e-6)
Code has docstrings on every method explaining the math
300-word writeup in README

Time estimate: 8-10 hours. This is the hardest project. The _best_split function alone takes most people 2-3 hours to get right. That’s fine. The confusion is the point.

Project 3 · Customer Churn Model — Full Pipeline, Deployed as API¶

What This Proves¶

You can take a business problem, engineer features from raw data, build and tune a model, package it properly, and deploy it as a working API with input validation and drift monitoring. This is the project you demo in interviews. It maps directly to what ML engineers do on their first week at a new job.

Project Spec¶

Repository: churn-prediction-api

Dataset: IBM Telco Customer Churn (free on Kaggle)

7,043 customers, 21 features, ~26.5% churn rate
Mix of numeric, categorical, binary features
Real business context: telecom churn prediction

Project Structure:
churn-prediction-api/
├── data/
│   └── telco_churn.csv
├── notebooks/
│   └── 01_eda_and_modeling.ipynb  ← full development notebook
├── src/
│   ├── features.py                 ← feature engineering functions
│   ├── train.py                    ← training script (saves model to models/)
│   ├── serve.py                    ← FastAPI application
│   └── monitor.py                  ← drift monitoring
├── models/                         ← saved .joblib + metadata.json
├── tests/
│   ├── test_features.py
│   └── test_api.py
├── requirements.txt
├── Dockerfile
└── README.md

Phase 1: EDA + Feature Engineering (2 hrs)

Required EDA findings (document all in notebook):
- Churn rate by Contract type (month-to-month vs. annual)
- Churn rate by tenure buckets
- Correlation between TotalCharges and tenure (near-perfect: r > 0.95)
  → Decision: drop TotalCharges or keep it? Document your reasoning.
- Distribution of MonthlyCharges for churned vs. retained customers

Required engineered features:
- tenure_bucket: binned into 5 groups (0-12, 12-24, 24-36, 36-48, 48-60, 60+)
- avg_monthly_charge: TotalCharges / (tenure + 1)  — captures spend rate
- has_multiple_services: sum of PhoneService, InternetService, OnlineSecurity,
  OnlineBackup, DeviceProtection, TechSupport (binary flags)
- is_senior_with_partner: SeniorCitizen == 1 AND Partner == "Yes"

Phase 2: Modeling (2 hrs)

Requirements:
1. Baseline: LogisticRegression (establishes minimum bar)
2. Production model: XGBoost or LightGBM with RandomizedSearchCV (50 iters)
3. CV: StratifiedKFold(5), scoring='roc_auc'
4. Class imbalance handling: try scale_pos_weight in XGBoost OR 
   class_weight='balanced' in sklearn models. Document effect on PR-AUC.
5. Calibration: apply isotonic regression calibration. Report Brier score
   before/after. Calibration matters here — the API outputs probabilities.
6. Final evaluation (on holdout test set, touched once):
   - ROC-AUC
   - PR-AUC  
   - Brier Score
   - Calibration curve plot
   - Classification report at threshold=0.5 and threshold=0.35
     (lower threshold because FN = lost revenue opportunity)

Required Results:

Test ROC-AUC ≥ 0.82 (achievable without feature leakage on this dataset)
Brier score ≤ 0.16 (after calibration)
Report shows PR curve and ROC curve side-by-side

Phase 3: API (2 hrs)

Implement the FastAPI server from 05_classical_ml_in_production.md, adapted for Telco churn:

Required endpoints:
GET  /health         → {"status": "ok", "model_version": "..."}
GET  /model/info     → metadata JSON
POST /predict        → single customer prediction
POST /predict/batch  → batch up to 200 customers

Required input validation (Pydantic):
- tenure: int, >= 0
- MonthlyCharges: float, >= 0
- Contract: str, must be one of ["Month-to-month", "One year", "Two year"]
- All other fields: type-checked with appropriate ranges

Required response:
- churn_probability: float [0, 1]
- churn_risk: "HIGH" (>0.6) | "MEDIUM" (0.3-0.6) | "LOW" (<0.3)
- model_version: str
- inference_time_ms: float

Phase 4: Testing + Monitoring (1 hr)

# tests/test_api.py (use pytest + httpx or requests)

def test_health_endpoint():
    response = requests.get("http://localhost:8000/health")
    assert response.status_code == 200
    assert response.json()["status"] == "ok"


def test_predict_valid_input():
    payload = {
        "tenure": 12,
        "MonthlyCharges": 65.0,
        "Contract": "Month-to-month",
        # ... all required fields
    }
    response = requests.post("http://localhost:8000/predict", json=payload)
    assert response.status_code == 200
    data = response.json()
    assert 0.0 <= data["churn_probability"] <= 1.0
    assert data["churn_risk"] in ["HIGH", "MEDIUM", "LOW"]


def test_invalid_contract_type_returns_422():
    payload = {"tenure": 12, "MonthlyCharges": 65.0, "Contract": "invalid_type"}
    response = requests.post("http://localhost:8000/predict", json=payload)
    assert response.status_code == 422  # FastAPI validation error, not 500


def test_batch_predict_limit():
    """Batch size > 200 should return 400."""
    payload = [{"tenure": 12, "MonthlyCharges": 65.0, "Contract": "Month-to-month"}
               for _ in range(201)]
    response = requests.post("http://localhost:8000/predict/batch", json=payload)
    assert response.status_code == 400


def test_inference_latency():
    """Single prediction must complete in < 50ms."""
    import time
    payload = { ... }  # valid payload
    start = time.time()
    response = requests.post("http://localhost:8000/predict", json=payload)
    latency_ms = (time.time() - start) * 1000
    assert latency_ms < 50, f"Latency {latency_ms:.1f}ms exceeds 50ms SLA"

Phase 5: Dockerfile (0.5 hrs)

FROM python:3.10-slim

WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

COPY src/ ./src/
COPY models/ ./models/

EXPOSE 8000
CMD ["uvicorn", "src.serve:app", "--host", "0.0.0.0", "--port", "8000"]

# Build and run
docker build -t churn-api:latest .
docker run -p 8000:8000 churn-api:latest

# Verify
curl http://localhost:8000/health

Acceptance Criteria:

Test ROC-AUC ≥ 0.82 on held-out test set (documented in notebook)
Calibration curve plotted + Brier score improved by isotonic regression
FastAPI server runs locally: all 4 required endpoints return correct responses
All 5 test cases pass (pytest tests/)
Invalid input returns 422, not 500
Single prediction latency < 50ms (measured in tests)
Docker image builds and runs cleanly
README includes: dataset, problem statement, model approach (2 paragraphs), how to run locally, API usage example with curl

Time estimate: 7-9 hours

Phase 1 Portfolio Summary¶

Project	Core Skills Demonstrated	Deliverable	Est. Hours
Kaggle Titanic + House Prices	EDA, feature engineering, model comparison, competition workflow	Leaderboard score + clean notebooks	10-12 hrs
Random Forest from Scratch	Algorithm internals, tree construction, ensemble theory, numpy	Passing test suite + technical writeup	8-10 hrs
Churn API	End-to-end pipeline, calibration, FastAPI, Docker, testing	Deployed API + passing test suite	7-9 hrs

Total estimated time: 25-31 hours across Month 4 (roughly 7-8 hrs/week on top of study time)

Where to publish:

All three projects: public GitHub repos with descriptive READMEs
Titanic/House Prices: link your Kaggle profile showing leaderboard position
Churn API: record a 2-minute Loom walkthrough of the live API — one video is worth fifty bullet points on a resume

Return to README.md · Previous: 05_classical_ml_in_production.md

06 · Phase 1 Projects¶

Project 1 · Kaggle Competition Progression: Titanic → House Prices¶

What This Proves¶

Project Spec¶

Portfolio Output¶

Project 2 · Random Forest from Scratch¶

What This Proves¶

Project Spec¶

Project 3 · Customer Churn Model — Full Pipeline, Deployed as API¶

What This Proves¶

Project Spec¶

Phase 1 Portfolio Summary¶