Phase 0 Projects: Proving You Know the Math

Phase 0 · Week 8 · ~12–15 hours total

Projects are the only honest test of understanding. Reading about eigendecomposition and implementing PCA from scratch are different cognitive activities — the second one reveals every gap the first one hides. These three projects are not exercises to be completed and forgotten. They are portfolio artifacts that demonstrate to any technical interviewer or collaborator that your mathematical foundations are operational, not decorative. Each project has explicit acceptance criteria: either you pass them or you don’t.

---

Project 1: PCA from Scratch in NumPy

What This Proves

You understand eigendecomposition of covariance matrices, the geometric meaning of principal components, variance explanation, and the connection between linear algebra and dimensionality reduction — without the abstraction of sklearn hiding the steps.

Specification

Dataset: UCI Wine Quality dataset (available via sklearn’s load_wine() — 178 samples, 13 features, 3 classes). Do not use a toy dataset; the 13-dimensional structure makes the variance explanation meaningful.

What to implement:

class PCAFromScratch:
    """
    PCA implementation using eigendecomposition of the covariance matrix.
    API must match sklearn's PCA interface exactly.
    """
    
    def __init__(self, n_components):
        self.n_components = n_components
        self.components_ = None           # shape: (n_components, n_features)
        self.explained_variance_ = None   # eigenvalues
        self.explained_variance_ratio_ = None
        self.mean_ = None
    
    def fit(self, X):
        """
        Compute principal components from data X.
        X: array of shape (n_samples, n_features)
        """
        # YOUR IMPLEMENTATION HERE
        # Step 1: Center data
        # Step 2: Compute covariance matrix (use np.cov, note the transposition)
        # Step 3: Eigendecomposition (use np.linalg.eigh for symmetric matrix)
        # Step 4: Sort by eigenvalue descending
        # Step 5: Store top-n_components eigenvectors as self.components_
        # Step 6: Compute explained variance ratio
        raise NotImplementedError
    
    def transform(self, X):
        """Project X onto principal components."""
        raise NotImplementedError
    
    def fit_transform(self, X):
        self.fit(X)
        return self.transform(X)
    
    def inverse_transform(self, X_reduced):
        """Reconstruct approximation of original data from reduced representation."""
        raise NotImplementedError

Required deliverables:

1. Correctness verification:

from sklearn.decomposition import PCA
from sklearn.datasets import load_wine

X, y = load_wine(return_X_y=True)

# Your implementation
pca_scratch = PCAFromScratch(n_components=2)
X_scratch = pca_scratch.fit_transform(X)

# Sklearn reference
pca_sklearn = PCA(n_components=2)
X_sklearn = pca_sklearn.fit_transform(X)

# Acceptance: match within float tolerance (signs may differ)
# Correct sign alignment:
for i in range(2):
    if not np.allclose(pca_scratch.components_[i], pca_sklearn.components_[i], atol=1e-5):
        # Try flipping sign (both are valid principal components)
        assert np.allclose(pca_scratch.components_[i], -pca_sklearn.components_[i], atol=1e-5), \
            f"Component {i} does not match sklearn (even with sign flip)"

print("✅ Components match sklearn")

# Explained variance ratio must match
assert np.allclose(pca_scratch.explained_variance_ratio_, 
                   pca_sklearn.explained_variance_ratio_, atol=1e-5)
print(f"✅ Explained variance ratio: {pca_scratch.explained_variance_ratio_}")

2. Scree plot: Plot explained variance ratio for all 13 components. Mark the elbow. State what percentage of variance is captured by the top 2 and top 5 components.

3. 2D visualization: Scatter plot of the 2-component projection, colored by wine class. The three classes should be visually separable.

4. Reconstruction error analysis: Compute ||X - X_reconstructed||_F (Frobenius norm) for k=1,2,3,5,8,13 components. Plot reconstruction error vs k.

Acceptance Criteria

• np.allclose(scratch_components, sklearn_components, atol=1e-5) passes (after sign adjustment)

• np.allclose(scratch_evr, sklearn_evr, atol=1e-5) passes

• Scree plot with labeled axes, elbow identified

• 2D scatter plot shows class separation

• Reconstruction error curve is monotonically decreasing (it must be — if not, your implementation has a bug)

• A written comment (3-5 sentences) explaining why PCA finds the directions it does (not just what it does)

Where to Share

GitHub repo named ml-foundations-from-scratch. This becomes a portfolio anchor for Phase 0. Tag it numpy, pca, linear-algebra, machine-learning.

Time estimate: 4–6 hours (2 hours implementation, 1 hour debugging, 1–2 hours visualization + write-up)

---

Project 2: Gradient Descent on a Loss Surface — Visualize Convergence

What This Proves

You understand gradients as geometric objects pointing toward steepest ascent, gradient descent as a sequence of locally-optimal steps, learning rate as a geometric parameter that controls step size, and loss landscapes as the actual terrain your optimizer navigates.

Specification

Part A — 2D Quadratic (Warm-up):

# Loss function: f(w0, w1) = (w0 - 2)² + 5(w1 + 1)²
# Gradient: [2(w0-2), 10(w1+1)]
# Global minimum: (2, -1)

# Implement vanilla gradient descent and visualize the trajectory
# Required: contour plot of loss surface with optimization path overlaid

Part B — Linear Regression Loss Surface:

def run_gradient_descent_experiment(X, y, lr, n_iter, w_init=None):
    """
    Run gradient descent and return full trajectory.
    
    Returns:
        w_history: array of shape (n_iter+1, n_params)
        loss_history: array of shape (n_iter+1,)
    """
    raise NotImplementedError


# Generate a 2-parameter regression problem
np.random.seed(42)
X_raw = np.random.randn(200, 1)
y = 3.0 * X_raw.ravel() + 1.5 + 0.5 * np.random.randn(200)
# Add bias column
X = np.hstack([X_raw, np.ones((200, 1))])  # shape (200, 2)
# True params: w=[3.0, 1.5]

Required experiments and plots:

1. Learning rate comparison: Run gradient descent with lr ∈ {0.0001, 0.001, 0.01, 0.1, 1.0} for 500 iterations. On a single figure, plot:

   • Row 1: Loss curves (log scale y-axis) for all 5 learning rates

   • Row 2: Last 100 steps of weight trajectory on loss contour for each lr

2. Convergence analysis:

   • For each learning rate, record: (a) iterations to converge (loss change < 1e-6), or “diverged”

   • Produce a table:

   Learning Rate	Final Loss	Iterations to Converge	Status
   0.0001	…	…	Slow
   0.001	…	…	OK
   0.01	…	…	Fast
   0.1	…	…	…
   1.0	…	…	Diverged

3. Momentum comparison: Implement gradient descent with momentum:

   v = 0
   v = momentum * v - lr * gradient
   w = w + v
   

   Compare convergence of plain GD vs momentum=0.9 on the same problem. They should converge in different numbers of steps.

Acceptance Criteria

• Loss curves produced for all 5 learning rates with labeled axes

• Loss decreases monotonically for at least lr ∈ {0.0001, 0.001, 0.01}

• lr=1.0 shows divergence (loss increases or oscillates) — if it doesn’t, recheck your gradient implementation

• Convergence table filled in with correct values

• Momentum implementation verified (converges faster than plain GD on the same lr)

• Written explanation (4–6 sentences): Why does lr=1.0 diverge? Frame the answer using the Hessian eigenvalue, not just “the step is too big.”

Where to Share

Add to same ml-foundations-from-scratch GitHub repo. Create a /notebooks/gradient_descent_viz.ipynb.

Time estimate: 3–4 hours

---

Project 3: Probability Calibration Exercise — Fit Distributions to Real Data

What This Proves

You understand probability distributions as models of data-generating processes (not just mathematical formulas), maximum likelihood estimation as a principled fitting procedure, and model comparison via KL divergence.

Specification

Dataset: Use the scipy.stats module and the following real datasets:

• Precipitation data: Use 365 days of daily rainfall amounts (generate from a mixture or load from a real weather API/dataset)

• Alternatively: Load sklearn.datasets.load_breast_cancer() and work with any continuous feature column

Tasks:

import scipy.stats as stats
import numpy as np
import matplotlib.pyplot as plt

# Task 1: Fit multiple distributions to a continuous feature
# Using breast cancer dataset, feature 0 (mean radius)
from sklearn.datasets import load_breast_cancer
X, y = load_breast_cancer(return_X_y=True)
data = X[:, 0]  # mean radius feature

# Fit these distributions using MLE (scipy.stats handles this):
distributions = {
    'Normal': stats.norm,
    'Log-Normal': stats.lognorm,
    'Gamma': stats.gamma,
    'Exponential': stats.expon,
}

fit_results = {}
for name, dist in distributions.items():
    # scipy .fit() returns MLE parameters
    params = dist.fit(data)
    fit_results[name] = params

# Task 2: Compare fits using log-likelihood (higher is better)
# and KL divergence from empirical distribution (lower is better)
def empirical_kl_from_fit(data, dist, params, n_bins=50):
    """
    Estimate KL(empirical || fitted) using histogram approximation.
    """
    counts, bin_edges = np.histogram(data, bins=n_bins, density=True)
    bin_centers = (bin_edges[:-1] + bin_edges[1:]) / 2
    bin_width = bin_edges[1] - bin_edges[0]
    
    # Empirical probabilities
    p_empirical = counts * bin_width
    p_empirical = p_empirical / p_empirical.sum()
    
    # Fitted distribution probabilities
    p_fitted = dist.pdf(bin_centers, *params) * bin_width
    p_fitted = p_fitted / p_fitted.sum()
    
    # KL divergence
    mask = (p_empirical > 0) & (p_fitted > 0)
    kl = np.sum(p_empirical[mask] * np.log(p_empirical[mask] / p_fitted[mask]))
    return kl

Required deliverables:

1. Fit comparison plot: For the chosen feature, plot the empirical histogram overlaid with each distribution’s PDF. 4 subplots, one per distribution.

2. Model comparison table:

Distribution	Log-Likelihood	KL Divergence	Rank
Normal	…	…	…
Log-Normal	…	…	…
Gamma	…	…	…
Exponential	…	…	…

3. MLE parameter verification: For the Normal distribution fit, verify that params[0] (mean) equals data.mean() and params[1] (std) equals data.std() within 1e-4. This verifies you understand MLE for Gaussian.

4. Calibration check: For logistic regression on the breast cancer dataset, plot a calibration curve (use sklearn.calibration.calibration_curve). Is the model well-calibrated? If not, apply CalibratedClassifierCV and re-plot.

Acceptance Criteria

• All 4 distributions fit successfully (no exceptions)

• MLE verification: abs(params_normal[0] - data.mean()) < 1e-4

• KL divergence computed and ranked correctly (lower = better fit)

• Best-fitting distribution identified and the geometric reason stated (e.g., “mean radius is right-skewed with a positive floor → log-normal is appropriate because…”)

• Calibration curve produced for logistic regression

• Written conclusion: 3–4 sentences explaining what it would mean for a production ML model to be poorly calibrated, and why KL divergence (not just accuracy) captures this failure

Where to Share

Add to ml-foundations-from-scratch repo. This demonstrates that you think probabilistically about data, not just algorithmically.

Time estimate: 3–5 hours

---

Portfolio Summary for Phase 0

After completing all three projects, you have:

Artifact	What It Demonstrates	Where
pca_from_scratch.py	Linear algebra → ML transformation	GitHub
gradient_descent_viz.ipynb	Calculus → optimization mechanics	GitHub
distribution_fitting.ipynb	Probability → model design	GitHub

This is your Phase 0 credential. It’s not a certificate — it’s reproducible code with documented results. When you move to Phase 1, these artifacts remain live proof that your mathematical foundations are implementation-tested, not just read about.

Commit message convention: [P0-P1] PCA from scratch: eigendecomposition verified against sklearn — always reference the phase and project number.

---

Return to README.md · Next Phase: 02_phase_1_classical_ml/README.md