06 — Phase 5 Projects

Three projects. One capstone that demonstrates the full MLOps stack. Two focused projects that prove specific production skills. All three are senior ML engineer signal — they demonstrate that you can build ML systems that operate reliably over time, not just models that score well in notebooks.

The difference between a portfolio project and a research project: a portfolio project has a public URL, a green CI history, and evidence that it runs. A research project has a Jupyter notebook and a comment that says “TODO: clean this up.”

Project 1 — Full MLOps Pipeline (HARD GATE #2)

This is the capstone project for Phase 5 and the second hard gate of the entire 13-month roadmap. It is not a tutorial walkthrough. You are building a complete, end-to-end ML system for a real problem — one that you designed, implemented, and shipped.

The Constraint That Makes It Real

Pick a problem you actually care about. Kaggle datasets are acceptable. Your own data is better. The problem domain does not matter — what matters is that you go from raw data to a deployed, monitored API with every layer of the stack in place. A credit default predictor, a spam classifier, a tabular churn model — all of these are adequate. A project where you “fine-tuned GPT” with no serving, no CI, no monitoring, and no DVC is not.

What You Are Building

data/
  raw/                    # DVC-tracked, never modified
  processed/              # DVC-tracked, generated by pipeline
  reference_window/       # DVC-tracked, for monitoring baseline
src/
  preprocess.py
  train.py
  evaluate.py
  serve/
    main.py               # FastAPI serving endpoint
models/                   # DVC-tracked model artifacts
reports/
  monitoring/             # Evidently reports (weekly)
tests/
  test_preprocessing.py
  test_model_behavior.py
  test_api.py
scripts/
  validate_data.py
  validate_model.py
dvc.yaml                  # Reproducible ML pipeline
.github/workflows/
  ml_pipeline.yml         # Full CI/CD
README.md                 # Architecture diagram + deployment instructions

Acceptance Criteria (Binary — either done or not)

  • DVC: dvc repro runs the full pipeline from raw data to evaluation metrics. At least 2 tracked data versions (simulate by adding rows to the dataset and rerunning). dvc push has been run and data is in remote storage.

  • Experiment tracking: MLflow or W&B shows at least 5 experiments with different hyperparameters. Each run logs: all params, all metrics, the model artifact, and the data version hash as a tag.

  • CI/CD: GitHub Actions workflow passes on every push to main. Workflow includes: lint + type check → unit tests + data validation → model performance validation (blocks if below threshold) → Docker build → deploy. Green CI history is visible in the repo.

  • Deployed API: model is accessible via a live URL. Acceptable platforms: HuggingFace Spaces (Gradio or FastAPI), Fly.io, Railway, Render, or similar. The URL must be in the README and must respond to curl requests.

  • Monitoring: Evidently monitoring report generated on a simulated production sample showing drift analysis. At least one feature shows detected drift (simulate by applying a distribution shift to the production sample). Report saved as HTML in reports/monitoring/.

  • Architecture diagram: in README.md — a diagram (Mermaid or drawn) showing all components and data flows.

  • GitHub Actions history: screenshot or link to green CI runs is in the README.

Time Estimate

3–4 weeks of focused work (10–15 hrs/week = 30–60 total hours). This is the capstone of the production phase. Do not rush it. A half-finished pipeline that has “some CI” and “sort of works” is not a hard gate pass.

Where to Showcase

  1. GitHub repo — must be public. README must be written for someone who has never heard of your project. Include: what problem you solved, what the architecture looks like, how to run it locally, and what you learned.

  2. LinkedIn post — write 200–300 words explaining the architecture decision you are most proud of (or the thing that broke most badly and what you learned). Attach a screenshot of the CI pipeline or the monitoring report. “Built a full MLOps pipeline” with no specifics gets ignored. “Discovered that my model degraded 12% AUC within 6 weeks due to seasonal feature drift in the credit_utilization column, detected by monitoring the PSI score weekly with Evidently” is the kind of specificity that gets noticed.

Project 2 — Serve a Quantized LLM with vLLM

A focused, time-bounded project that proves you understand LLM serving infrastructure — not just API wrappers.

What You Are Building

Deploy a 7B parameter quantized model (Mistral-7B-Instruct-v0.3 or LLaMA-3.1-8B with AWQ or GPTQ quantization) using vLLM. Benchmark it. Compare it to the naive HuggingFace baseline. Compute the cost per million tokens at your serving setup. Write it up.

VRAM Requirements

7B model at 4-bit AWQ quantization: approximately 4–5GB VRAM. The A10G (24GB) on RunPod handles this comfortably at 20+ concurrent users. The T4 (16GB) on Colab Pro+ handles it at lower concurrency. A100 (40GB) handles the 13B version at 4-bit.

Acceptance Criteria

  • Serving endpoint live: python -m vllm.entrypoints.openai.api_server ... running and responding to requests. OpenAI-compatible API verified with a Python client.

  • Benchmark report: a table with the following measurements, collected using the benchmark script from 02_model_serving_and_inference.md or equivalent:

    Concurrency

    Req/s

    Tok/s

    p50 Latency

    p95 Latency

    p99 Latency

    1

    5

    10

    20

  • Comparison: vLLM throughput vs. model.generate() naive HuggingFace throughput on the same hardware. State the speedup factor. It should be somewhere in the 5–15x range on a consumer GPU.

  • Cost calculation:

    • Your GPU cost per hour (e.g., RunPod A10G: $0.30/hour)

    • Your measured throughput in tokens/second at steady load (e.g., 280 tok/s)

    • Cost per 1M tokens = (1,000,000 / (throughput_tok_s × 3600)) × cost_per_hour × 1,000,000

    • Compare to OpenAI GPT-4o-mini pricing ($0.15/1M input tokens as of 2025) — is self-hosting cheaper for your use case at what volume?

Time Estimate

8–12 hours. Most of this is setup time (RunPod instance, CUDA drivers, vLLM installation) and benchmark running time. The actual code is minimal.

What This Proves

You understand LLM serving infrastructure at the level of someone who has actually run it — not just called an API. You can quantify the performance vs. cost tradeoff, which is the question every team shipping LLM features to production has to answer.

Project 3 — Model Monitoring in Production

A project that proves you understand the full lifecycle of a deployed model — not just the launch, but the ongoing health maintenance.

What You Are Building

Deploy a model (any classification or regression problem — reuse the Project 1 model if you like), let it run against simulated production traffic, detect and alert on a data drift event, and write a formal incident report.

Simulating Production

You do not need real users. You need a way to generate batches of production data that you can run your monitoring against. Two valid simulation approaches:

Approach A — Temporal simulation: the training data is from one time period; the “production” data is from a later time period with different statistics. Many Kaggle datasets have timestamps. Split by time: train on the first 70% chronologically, treat the last 30% as “production arriving over 6 weeks.”

Approach B — Explicit distribution shift: take a sample from the test set, then create a shifted version by: multiplying a key feature by a factor (income × 1.5 to simulate inflation), adding Gaussian noise (μ=0, σ=0.3×std), or swapping categorical proportions. The first batch is “week 1 production” (unshifted), the second batch is “week 6 production” (shifted). Run Evidently on both.

Acceptance Criteria

  • Model deployed and serving: predictions are being served (locally via FastAPI, or on a free platform). Predictions are being logged with timestamps, input features, and model version.

  • Monitoring scheduled: Evidently report runs on a schedule or is triggered by new data arrival. At minimum, you have a script that can be run manually and generates a report. For bonus points: a GitHub Actions cron job that runs it weekly.

  • Drift event detected and documented: at least one drift event is detected (PSI > 0.2 or KS p-value < 0.05 on at least one key feature). The monitoring report shows this clearly.

  • Alert mechanism: one of the following is implemented:

    • Email via SMTP (Python smtplib)

    • Slack webhook (requests.post to a Slack Incoming Webhook URL — free to set up)

    • GitHub issue (use the GitHub API to open an issue when drift is detected)

  • Incident report written (in reports/incident_YYYY-MM-DD.md):

    ## Drift Incident Report — [Date]

**Detected by**: Evidently PSI monitoring (weekly batch job)
**Detection date**: [date]
**Feature(s) affected**: [feature name(s)]
**Drift metric**: PSI = [value] (threshold: 0.2)

### Root Cause
[What caused the distribution shift. For simulated drift: "Simulated by multiplying
annual_income by 1.5 to represent an inflationary scenario."]

### Impact Assessment
Model AUC on the drifted production window: [value] vs. training AUC: [value]
Estimated impact: [X]% degradation in prediction accuracy

### Resolution
[Retrained model on updated data. New model AUC: [value]. Deployed to production on [date].]

### Lessons Learned
[One concrete thing you would do differently in a real production system.]

Time Estimate

10–15 hours.

What This Proves

You understand that model deployment is not a one-time event. You have operational experience with the monitoring-alert-investigate-retrain cycle. The incident report, in particular, is documentation that a senior engineer would write — it demonstrates that you can communicate technical findings to stakeholders, not just fix the code and move on.

Portfolio Summary: What These Three Projects Signal

When a hiring manager or technical interviewer sees these three projects together, they map to specific capabilities:

Project

Signals

Equivalent Experience

Full MLOps Pipeline

Level 2 MLOps maturity, end-to-end systems thinking, CI/CD discipline

1–2 years production ML experience

vLLM Benchmarking

LLM infrastructure knowledge, quantitative benchmarking, cost analysis

LLM Eng at a company that self-hosts models

Production Monitoring

Operational ML mindset, incident response, statistical literacy

ML Platform or MLOps engineer

Most candidates have the first half of the ML lifecycle (training, evaluation, model architecture). These projects demonstrate the second half (deployment, monitoring, iteration). In 2025, with 55% of companies citing lack of MLOps practices as their primary ML deployment obstacle, the second half is the actual job.

Return to [README.md] · Next: [../07_phase_6_research_and_mastery/README.md]