Rung 5: LLM Engineering¶

Month 10 | Phase 4: LLMs | ~50–65 hours total ⚠️ HARD GATE #1 — Do not proceed to Phase 5 until this rung is complete and publicly verifiable.

This is the first gate because LLM engineering is now the most frequently faked skill in applied ML resumes. The gate exists to draw a hard line between candidates who have used the OpenAI API (everyone) and candidates who have actually fine-tuned a model, built a retrieval-augmented system over a real corpus, implemented a systematic evaluation protocol, and deployed something a stranger can use. The gate question is binary: “Can I show someone a live demo right now?” If the answer is no, the gate is not passed.

Why This Is a Hard Gate¶

By 2025, “I worked with LLMs” on a resume is meaningless without qualification. The minimum bar that distinguishes a genuine LLM engineer from an API consumer:

Fine-tuning: have you modified model weights for a specific task? (Not prompt engineering, not in-context learning)
RAG: have you built a retrieval system over a real document corpus? (Not a toy example, not a tutorial corpus)
Evaluation: have you implemented systematic, quantitative evaluation of your system? (Not “it seems to work well”)
Deployment: is it accessible to an external user right now? (Not “I ran it locally”)

All four must be true for this gate to open.

What to Build¶

You are building two interconnected systems that together demonstrate production-adjacent LLM engineering:

System 1: Domain-Specific Fine-Tuned LLM (QLoRA)¶

Fine-tune a 7B parameter model using QLoRA (Quantized Low-Rank Adaptation) on a specific domain task you can articulate clearly.

Model selection (any of these are acceptable):

Mistral-7B-v0.1 / Mistral-7B-Instruct-v0.3
Llama-3-8B / Llama-3.1-8B-Instruct
Qwen2-7B-Instruct
Gemma-2-9B-IT

Domain task — choose one with genuine specificity:

Code generation: Python data science code generation from natural language descriptions (fine-tune on filtered CodeSearchNet or Stack Overflow Python Q&A)
Technical QA: fine-tune on a specific technical domain relevant to your Zoho work (CRM documentation, SaaS technical writing, API documentation) — strongly preferred because it demonstrates business context
Medical / scientific summarization: fine-tune on PubMed abstracts → structured summaries
Indian language instruction following: fine-tune on a Tamil or Hindi instruction-following dataset (demonstrates domain awareness relevant to your context)
SQL generation: fine-tune on text-to-SQL for a specific schema type (enterprise, financial, e-commerce)

The “why this domain” question will be asked in every interview. Have a 2-sentence answer that is not “because there was a dataset for it.”

QLoRA setup — exact specification:

# Required libraries:
# transformers >= 4.40
# peft >= 0.10
# bitsandbytes >= 0.42
# trl >= 0.8
# datasets >= 2.18

# Quantization config (4-bit NF4):
from transformers import BitsAndBytesConfig
bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16,
    bnb_4bit_use_double_quant=True,  # nested quantization
)

# LoRA config — document why you chose these values:
from peft import LoraConfig
lora_config = LoraConfig(
    r=16,              # rank — explain the rank/capacity tradeoff
    lora_alpha=32,     # scaling factor — explain why alpha=2r is common
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj",
                    "gate_proj", "up_proj", "down_proj"],
    lora_dropout=0.05,
    bias="none",
    task_type="CAUSAL_LM",
)

Every hyperparameter in your LoRA config must be documented with a comment explaining what it controls and why you chose that value.

VRAM requirements and compute options:

7B model in 4-bit: ~6–8 GB VRAM — fits on Kaggle GPU (16GB T4), Colab Pro (A100 40GB), or RunPod (~$0.20/hr for A100)
Training time estimate: 2–4 hours for 1-3 epochs on a well-sized dataset (~10K–50K examples)
If you have access to a local machine with an RTX 4090 (24GB): this is your best option for iteration speed
Kaggle is the recommended free option: 2 T4 GPUs, 30hrs/week, sufficient for this task

System 2: RAG System Over a Real Corpus¶

Build a Retrieval-Augmented Generation pipeline over a corpus that is not a tutorial dataset. The corpus must have ≥ 500 documents with realistic heterogeneity (varying lengths, formats, technical depth).

Corpus options:

A curated set of technical documentation (Zoho API docs, Python library docs, a scientific paper collection in your domain)
A legal or policy document set (open government documents, license agreements)
A domain-specific FAQ/knowledge base you construct yourself (this is most impressive — it demonstrates problem definition skills)
arXiv papers in a specific subfield (download 500+ using the arXiv API)

RAG pipeline components — all must be implemented:

[User Query]
     ↓
[Query Preprocessing]       ← query cleaning, optional query expansion
     ↓
[Embedding Model]           ← use sentence-transformers or a HF embedding model
     ↓                       (document: BAAI/bge-m3 or UAE-Large-V1 recommended)
[Vector Store]              ← use ChromaDB, FAISS, or Qdrant locally
     ↓
[Top-k Retrieval]           ← k=3 to 5, document what k you chose and why
     ↓
[Context Construction]      ← how you format retrieved chunks into the prompt
     ↓
[LLM Generation]            ← your fine-tuned model OR a capable base model
     ↓
[Response + Citations]      ← source documents must be returned with response
     ↓
[Evaluation]                ← RAGAS or equivalent

Chunking strategy must be documented: what chunk size (tokens), what overlap, why. “512 tokens with 50 overlap” with no explanation is not documentation. “512 tokens because it fits the context window of the embedding model and is small enough to preserve semantic coherence without excessive fragmentation; 50-token overlap to prevent cutting mid-sentence across chunk boundaries” is documentation.

Evaluation Protocol (Non-Negotiable)¶

Subjective evaluation (“it seems good”) does not pass this gate. You must implement systematic evaluation using at minimum the following:

For the Fine-Tuned Model:¶

Metric	How to Compute	What It Measures
Perplexity on held-out test set	`torch.exp(model_loss)`	Language modeling quality
Task-specific metric (F1, BLEU, ROUGE)	Task-appropriate; document your choice	Downstream task performance
Zero-shot vs. fine-tuned delta	Compare fine-tuned vs. base model	Quantify the fine-tuning benefit
Human evaluation (20 examples minimum)	Rate on 1–5 scale: relevance, accuracy, fluency	Subjective quality floor

For the RAG System:¶

Use RAGAS (Retrieval Augmented Generation Assessment) or implement equivalent metrics manually:

Metric	What It Measures	Target Threshold
Context Recall	Are relevant documents being retrieved?	≥ 0.70
Context Precision	Are retrieved docs actually relevant?	≥ 0.65
Answer Faithfulness	Does the answer stay grounded in context?	≥ 0.75
Answer Relevancy	Does the answer address the question?	≥ 0.70

Create a test set of ≥ 50 question-answer pairs for evaluation. Document how you created this test set (manual annotation? GPT-4 generated? automatic extraction?) because the test set creation methodology is part of the evaluation rigor.

Deployment Target¶

The system must be live and accessible to complete this gate.

Recommended platform: HuggingFace Spaces (Gradio)

Free tier supports CPU inference; use your quantized model (GGUF format via llama.cpp for CPU deployment, or quantize further to 2-bit for Space compatibility)
Alternatively: RunPod serverless function (pay-per-use, no idle cost) with a Gradio frontend on Spaces pointing to the RunPod endpoint
The demo must include: text input, response output, and displayed source citations from RAG

Minimum demo requirements:

A stranger can visit the URL and ask a question
The system returns a coherent answer with source attribution
The demo URL is stable (not a temporary Colab link that expires)
Average inference latency ≤ 8 seconds for a response (document this in your README)

Time Estimate¶

Task	Estimated Hours
Domain selection and dataset preparation	5–8
QLoRA fine-tuning setup, first run, debugging	8–10
Training run(s) and hyperparameter iteration	6–8
RAG pipeline implementation	8–10
Corpus processing and embedding	3–4
RAGAS evaluation implementation and reporting	4–5
Deployment (Spaces + model card)	4–5
Documentation (README + evaluation report)	4–5
Total	42–55 hours

This maps to roughly 5–6 weeks of Phase 4 at target pace. Month 10 is when this is due. Start the domain selection and dataset preparation in Month 9 so Month 10 is focused on implementation.

What Weakens This Rung¶

Generic domain, no justification: fine-tuning on “general instruction following” when a specific domain would be more defensible. The domain choice is a signal about whether you thought about the problem or just picked the nearest dataset.
Evaluation on the training distribution: if your test set came from the same source as your training data without held-out split, your metrics are optimistic fiction. Document your data split rigorously.
RAG with no chunking strategy explanation: chunking is where most RAG systems fail silently. If you didn’t think about it, the system quality reflects that.
Broken or inaccessible demo at interview time: if the first thing an interviewer does is click your demo link and it 404s, the conversation has already gone badly. Test the link the morning of every interview.
Model card with empty “limitations” section: every LLM has hallucinations, biases, and domain coverage gaps. A model card that claims no limitations is a model card that wasn’t written by someone who ran the model.
Pinned requirements at wildly incompatible versions: if your requirements.txt specifies transformers==4.28.0 but peft==0.10.0 requires transformers>=4.36, the setup fails on first pip install. Test it in a clean environment.

Return to README.md · Previous: 04_rung_4_transformer_lab.md · Next: 06_rung_6_production_ml_system.md