Rung 6: Production ML System¶

Month 12 | Phase 5: MLOps | ~55–75 hours total ⚠️ HARD GATE #2 — Do not sit for senior-level ML interviews until this system is live, monitored, and demonstrable.

This rung exists because of a specific, observable gap in the applied ML landscape: researchers who can build brilliant models that break in production, and engineers who can ship but not learn. The second hard gate is placed here because it tests a fundamentally different capability than every previous rung — not whether you understand ML, but whether you can build an ML system that survives contact with real infrastructure, real data drift, and real operational requirements. By 2025, 70%+ of senior MLE job descriptions in India explicitly require end-to-end ML pipeline experience and MLOps fluency. This rung is the proof.

Why This Is a Hard Gate¶

Production ML systems are the single hardest thing to fake in an interview. A hiring manager can ask: “Walk me through your monitoring dashboard right now.” If you cannot do it, the conversation is over. If you can, the conversation deepens into the kind of technical exchange that leads to offers. The gate is not about the technology stack — it is about the discipline of building something that must keep working after you stop paying attention to it.

What to Build: The Minimum Viable Production ML System¶

You are building a complete, live ML pipeline that moves a model from training to serving with monitoring. The system does not need to be impressive in scope — it needs to be complete in architecture. A simple model with a proper production setup demonstrates far more engineering maturity than a complex model with no monitoring.

Core Architecture¶

┌─────────────────────────────────────────────────────────────────┐
│                    PRODUCTION ML SYSTEM                         │
│                                                                  │
│  ┌──────────┐    ┌───────────┐    ┌──────────┐   ┌──────────┐  │
│  │  Data    │───▶│Experiment │───▶│  Model   │──▶│ Serving  │  │
│  │Versioning│    │ Tracking  │    │ Registry │   │  Layer   │  │
│  │  (DVC)   │    │  (MLflow) │    │(MLflow/  │   │(FastAPI/ │  │
│  └──────────┘    └───────────┘    │ HF Hub)  │   │BentoML)  │  │
│                                   └──────────┘   └────┬─────┘  │
│  ┌──────────────────────────────────────────────────────────┐   │
│  │                   CI/CD Pipeline (GitHub Actions)        │   │
│  │  test → lint → train → evaluate → register → deploy     │   │
│  └──────────────────────────────────────────────────────────┘   │
│                                                       │          │
│                                              ┌────────▼─────┐   │
│                                              │  Monitoring  │   │
│                                              │  (Evidently/ │   │
│                                              │   Grafana)   │   │
│                                              └──────────────┘   │
└─────────────────────────────────────────────────────────────────┘

Every box in this diagram must be implemented and working. A system missing monitoring is not a production system. A system missing CI/CD is a manual deployment. A system missing data versioning cannot be reproduced or debugged when something breaks.

Required Components¶

1. Data Versioning with DVC¶

Initialize DVC in your repository
Track your training dataset with dvc add — data is not committed to Git, only the .dvc pointer file
Configure a remote storage target: DagsHub (free, integrates with MLflow), Google Drive, or S3 (free tier)
Create at least 2 versions of your dataset: an initial version and a “drift-introduced” version (add 10–15% of data from a different distribution)
Document: dvc repro should reproduce your training run from scratch

2. Experiment Tracking with MLflow¶

Every training run must log:

with mlflow.start_run(run_name=f"experiment_{datetime.now().strftime('%Y%m%d_%H%M%S')}"):
    mlflow.log_params({
        "model_type": ...,
        "n_estimators": ...,   # or equivalent
        "learning_rate": ...,
        "feature_set_version": ...,   # from DVC
        "train_data_hash": ...,       # SHA256 of training data
    })
    
    mlflow.log_metrics({
        "train_f1": ...,
        "val_f1": ...,
        "val_precision": ...,
        "val_recall": ...,
        "train_time_seconds": ...,
    })
    
    mlflow.log_artifact("figures/confusion_matrix.png")
    mlflow.sklearn.log_model(model, "model")  # or mlflow.pytorch / mlflow.transformers

Run at least 5 experiments with different configurations. The MLflow UI showing experiment comparison is a critical demo artifact — screenshot it for your README.

3. Model Registry and Promotion¶

Register your best model in MLflow Model Registry with stages: Staging → Production
Include a promotion script that: runs evaluation on a validation set, checks performance against a minimum threshold, and promotes to Production only if threshold is met
Document the threshold: e.g., “model must achieve F1 ≥ 0.82 to be promoted to Production”
Include a rollback mechanism: script that can revert Production model to the previous registered version in under 2 minutes

4. Serving Layer¶

Option A: FastAPI REST endpoint (recommended)

from fastapi import FastAPI
from pydantic import BaseModel
import mlflow

app = FastAPI()
model = mlflow.pyfunc.load_model("models:/YourModel/Production")

class PredictionRequest(BaseModel):
    features: list[float]  # or dict for named features
    
class PredictionResponse(BaseModel):
    prediction: float | str
    probability: float | None
    model_version: str
    inference_latency_ms: float

@app.post("/predict", response_model=PredictionResponse)
async def predict(request: PredictionRequest):
    # ... implementation

The API must include: input validation (Pydantic), prediction, probability/confidence, model version in response, latency logging
Deploy to: Railway (free tier), Render (free tier), or Hugging Face Spaces (Docker option)
The endpoint must be public and documented in a Swagger UI (/docs)

Option B: BentoML — if you prefer a more ML-native serving framework. Document why you chose it.

5. CI/CD Pipeline (GitHub Actions)¶

Your .github/workflows/ml_pipeline.yml must trigger on push to main and execute:

jobs:
  test:
    - pytest tests/ (unit tests for feature engineering + data validation)
  
  train:
    - dvc repro (reproduce training from current data version)
    
  evaluate:
    - Compare new model performance vs. registered Production model
    - Fail pipeline if degradation > 2% on primary metric
    
  register:
    - mlflow.register_model() if evaluation passes
    
  deploy:
    - Deploy to serving infrastructure if on main branch
    - Send notification (Slack webhook or email) with metrics summary

This pipeline must have run at least 3 times with success — visible in GitHub Actions history. One forced failure (intentionally breaking a test to show the pipeline catches it) is even better.

6. Monitoring¶

This is where the most portfolios stop — and where the most hiring managers look. Implement monitoring using Evidently AI (free, open source):

Data drift monitoring:

from evidently.report import Report
from evidently.metric_preset import DataDriftPreset, DataQualityPreset

# Run on schedule (weekly) or on each new batch
report = Report(metrics=[DataDriftPreset(), DataQualityPreset()])
report.run(reference_data=train_df, current_data=new_batch_df)
report.save_html("monitoring/drift_report.html")

Prediction monitoring (model performance drift):

Track: prediction distribution over time, confidence score distribution, feature importance stability
For classification: track precision/recall/F1 on labeled batches (even small ones — 50 examples/week)

What must be demonstrable:

A monitoring report generated from your “drift-introduced” dataset version showing detected drift
A README section titled “Monitoring” with a screenshot of the Evidently report showing drift detection

Project Choice¶

The ML task underlying this system should be:

Real enough to have genuine data challenges (not a toy dataset)
Small enough to train in < 30 minutes on free hardware
Interpretable enough to have explainable predictions

Strong choices for this rung:

Customer churn prediction (tabular, business-relevant, clearly scoped)
Text classification relevant to your domain at Zoho (support ticket categorization, email intent classification)
Demand/sales forecasting on a public retail dataset
Anomaly detection in a system metrics or network traffic dataset

Do not choose a project that requires large GPU compute for training — that makes CI/CD impractical. The system architecture is the point, not the model sophistication.

Time Estimate¶

Task	Estimated Hours
Project selection and baseline model	4–6
DVC setup and data versioning	4–5
MLflow integration and experiment tracking	5–6
Model registry and promotion logic	3–4
FastAPI serving layer	5–6
Deployment (Railway/Render)	3–4
GitHub Actions CI/CD pipeline	6–8
Evidently monitoring setup	5–6
Documentation (README, runbook, architecture diagram)	5–6
End-to-end testing and debugging	5–8
Total	45–59 hours

Budget 3–4 weeks of focused work in Month 12. This is the highest-effort rung in the portfolio. It is also the rung that most directly demonstrates senior ML engineering capability.

What Weakens This Rung¶

Demo mode only, no live infrastructure: “I built this but it’s not running right now” fails the gate. The system must be live at interview time, not reconstructable.
CI/CD that never runs: a workflow.yml file with no run history is a configuration file, not a pipeline. The Actions tab must show evidence of actual runs.
Monitoring reports that are static screenshots from training time: monitoring is live, not historical. The demonstration must show what happens when you feed new data to the monitoring system.
No runbook: if you can’t explain in 5 commands how to retrain and redeploy, you haven’t built a system that anyone else could operate.
requirements.txt with unpinned versions: fastapi>=0.100 on a production service is a liability. Pin everything. Document why specific versions were chosen if there were conflicts.
System architecture diagram missing: this is the single document most interviewers want to see first. Draw it. ASCII art in markdown is acceptable. No diagram = you haven’t thought about the system holistically.
Overengineering the model, underengineering the infrastructure: a Gradient Boosted ensemble with 50 features and no monitoring is weaker than a logistic regression with a complete MLOps stack. The point of this rung is the infrastructure, not the model.

Return to README.md · Previous: 05_rung_5_llm_engineering.md · Next: 07_rung_7_paper_reproduction.md