Phase 4 Projects: Generative AI Frontier

These are not tutorial exercises. They are the three portfolio artifacts that demonstrate you have internalized the material in this phase and can operate at the boundary of what’s deployed in production today. Each project has a concrete specification, defined acceptance criteria, and a clear signal to a hiring committee or technical reviewer. Do not proceed to Phase 5 until at least two of these three are complete and documented.

Project 1: Domain-Specific QLoRA Fine-Tune with Rigorous Evaluation

Goal: Fine-tune a 7B parameter model on a domain-specific dataset using QLoRA. Evaluate the result quantitatively (BLEU/ROUGE, perplexity) and qualitatively (human evaluation rubric). Demonstrate the trained model outperforms the base model on your target task.

Specification

Dataset requirements:

  • Minimum 1,000 instruction-response pairs in your target domain

  • Recommended domains (high-signal, real evaluation possible): legal clause extraction, medical QA (from PubMedQA), code generation in a specific language/framework, customer support for a specific product vertical

  • Format: Alpaca (instruction / input / output) or ChatML (system/user/assistant)

  • Split: 80% train / 10% validation / 10% test. The test split is held out until final evaluation — no peeking.

Model requirements:

  • Base: meta-llama/Llama-3.2-3B-Instruct (fits on 8GB VRAM with QLoRA) or mistralai/Mistral-7B-Instruct-v0.3 (requires 12GB VRAM)

  • Method: QLoRA (4-bit NF4 + LoRA r=16, alpha=32, dropout=0.05)

  • Target modules: q_proj, v_proj, k_proj, o_proj

VRAM requirements:

Model

QLoRA VRAM

Full Fine-Tune VRAM

Cloud cost estimate (A100 40GB)

Llama-3.2-3B

~6GB

~24GB

~$2-4/hour via Lambda Labs

Mistral-7B

~10GB

~56GB

~$2-4/hour, need A100 80GB

LLaMA-3.1-8B

~12GB

~64GB

~$2-4/hour, need A100 80GB

Training configuration:

# Expected training time: 2-4 hours for 1K examples, 3 epochs on A100 40GB
training_args = TrainingArguments(
    num_train_epochs=3,
    per_device_train_batch_size=4,
    gradient_accumulation_steps=4,  # Effective batch = 16
    learning_rate=2e-4,
    lr_scheduler_type="cosine",
    warmup_ratio=0.03,
    bf16=True,
    logging_steps=10,
    eval_steps=100,
    save_steps=100,
    load_best_model_at_end=True,
    metric_for_best_model="eval_loss"
)

Evaluation Protocol

Automated metrics (run on held-out test set):

from rouge_score import rouge_scorer
from nltk.translate.bleu_score import corpus_bleu
import evaluate

# ROUGE-L F1 score (measures recall + precision of n-gram overlap)
scorer = rouge_scorer.RougeScorer(['rougeL'], use_stemmer=True)
rouge_scores = [scorer.score(ref, pred)['rougeL'].fmeasure 
                for ref, pred in zip(references, predictions)]
print(f"ROUGE-L: {sum(rouge_scores)/len(rouge_scores):.3f}")

# Perplexity (lower = better language model fit to domain)
# Target: fine-tuned perplexity < 0.7 × base model perplexity on domain data

# BERTScore (semantic similarity, more meaningful than BLEU for open generation)
bertscore = evaluate.load("bertscore")
results = bertscore.compute(predictions=predictions, references=references, lang="en")
print(f"BERTScore F1: {sum(results['f1'])/len(results['f1']):.3f}")

Human evaluation rubric (score each 1-5):

  1. Accuracy — Is the response factually correct given the domain?

  2. Completeness — Does it address all aspects of the question?

  3. Format adherence — Does it follow the expected output format?

  4. Conciseness — Is there unnecessary verbosity?

  5. Hallucination — Does the model assert things not in the question or retrievable facts?

Evaluate 50 random test examples. Compute mean per dimension.

Acceptance Criteria

  • ROUGE-L ≥ 0.35 on domain test set (base model ROUGE-L on same test set measured as baseline)

  • Fine-tuned model ROUGE-L ≥ 1.2× base model ROUGE-L (at minimum 20% improvement)

  • Fine-tuned perplexity ≤ 0.75× base model perplexity on domain data

  • Human eval mean score ≥ 3.5/5 across all dimensions

  • Training loss curve shows convergence (no upward divergence on validation)

  • Merged model runs inference correctly: model.generate() produces coherent domain-specific output

  • Full reproducibility: training script, dataset, hyperparameters, random seed documented

Portfolio Signal

This demonstrates: QLoRA implementation proficiency, evaluation methodology, domain adaptation, resource-efficient training. The evaluation protocol — specifically the comparison against baseline and the human rubric — is what separates this from “I ran some code” to “I did an experiment.”

Project 2: Production-Grade RAG System with RAGAS Evaluation

Goal: Build a complete RAG pipeline over a real document corpus (not a toy dataset). Measure retrieval quality and answer quality with RAGAS. Optimize at least two RAG components (chunking strategy, embedding model, or reranker) and show quantified improvement.

Specification

Document corpus requirements:

  • Minimum 50 documents, minimum 500 pages total

  • Recommended: company annual reports, technical manuals, academic papers in a domain, legal documents, or any corpus where answers are verifiable

  • Must be PDFs or structured text — not arbitrary web scrapes

Pipeline architecture:

PDF Corpus
    ↓
Text Extraction (PyMuPDF / pdfplumber)
    ↓
Chunking (implement 3 strategies, compare)
    ↓
Embedding (implement 2 models, compare)
    ↓
Vector Store (Qdrant — self-hosted, or pgvector in Postgres)
    ↓
Retrieval (k=5 initial candidates)
    ↓
Reranking (cross-encoder: BAAI/bge-reranker-v2-m3)
    ↓
Generation (GPT-4o-mini or local Mistral-7B)
    ↓
RAGAS Evaluation

Chunking strategies to compare:

  1. Fixed-size: 512 tokens, 50-token overlap

  2. Semantic: sentence-transformers-based semantic splitting

  3. Hierarchical: parent-child (large parent chunk for context, small child chunk for precision)

Embedding models to compare:

  1. text-embedding-3-small (OpenAI, 1536d, $0.02/1M tokens)

  2. BAAI/bge-large-en-v1.5 (local, 1024d, free, MTEB 64.2)

RAGAS evaluation setup:

from ragas import evaluate
from ragas.metrics import (
    faithfulness,
    answer_relevancy,
    context_precision,
    context_recall
)
from datasets import Dataset

# Build evaluation dataset: 50 question-answer pairs
# Questions must be answerable from the corpus
# Ground truth answers must be extractable from corpus text
eval_dataset = Dataset.from_dict({
    "question": questions,        # List[str]
    "answer": generated_answers,  # List[str] — model output
    "contexts": retrieved_chunks, # List[List[str]] — what was retrieved
    "ground_truth": true_answers  # List[str] — verified correct answers
})

results = evaluate(eval_dataset, metrics=[
    faithfulness,
    answer_relevancy,
    context_precision,
    context_recall
])

print(results)
# Target scores:
# faithfulness:      > 0.85 (model doesn't hallucinate beyond context)
# answer_relevancy:  > 0.80 (answer addresses the question)
# context_precision: > 0.70 (retrieved chunks are relevant)
# context_recall:    > 0.65 (relevant chunks are being retrieved)

Optimization experiment: Run the full pipeline with each combination: 3 chunking × 2 embedding = 6 configurations. Record RAGAS scores for each. Document which configuration wins and why.

Acceptance Criteria

  • All 6 pipeline configurations evaluated with RAGAS

  • Best configuration achieves: faithfulness > 0.85, answer_relevancy > 0.80

  • Improvement from worst to best configuration ≥ 15% on context_precision

  • Reranker measurably improves context_precision (before/after reranker comparison)

  • System handles queries that have no answer in the corpus gracefully (returns “not found” not hallucination)

  • Evaluation dataset of ≥ 50 QA pairs documented with ground truth sources

  • Full pipeline runnable end-to-end from a single python run_rag.py --query "..." command

Portfolio Signal

This is the project that demonstrates you understand RAG as an engineering problem, not a tutorial. The multi-configuration experiment and RAGAS-based evaluation are what differentiate senior engineers from junior ones. The “no answer” handling is a reliability signal.

Project 3: Reliable Multi-Step Agent with Measurable Success Rate

Goal: Build a multi-step agent that solves a well-defined real task. Measure task completion rate over 100 runs. Achieve ≥ 80% success rate on the defined task. Implement production reliability patterns (circuit breaker, budget enforcement, structured outputs).

Specification

Task options (pick one with verifiable correctness):

Option A — Research Synthesis Agent: Input: A research question (e.g., “What are the three most cited papers on attention mechanisms in NLP, and what did each contribute?”) Output: Structured report with citations and summaries Success criterion: Output contains ≥ 3 correctly cited papers with accurate summaries (verify against ground truth)

Option B — Code Analysis Agent: Input: A GitHub repository URL Output: Security vulnerability report + complexity analysis Success criterion: Report identifies all intentionally planted vulnerabilities in a test repo (use OWASP WebGoat or similar)

Option C — Data Processing Agent: Input: A CSV file with specified transformations needed (e.g., “normalize all monetary columns, fill missing dates with forward-fill, remove rows where revenue < 0”) Output: Transformed CSV + transformation log Success criterion: Output matches a pre-computed reference transform on 10 test files

Required agent components:

# Mandatory production patterns to implement:

# 1. Budget enforcer (hard limit)
class AgentBudget:
    max_steps: int = 15
    max_tokens: int = 50_000
    max_wall_time_seconds: int = 120
    current_cost_usd: float = 0.0
    max_cost_usd: float = 0.50  # Hard cap per run

# 2. Structured decision interface
class AgentDecision(BaseModel):
    thought: str
    next_action: Literal["search", "calculate", "read_file", "write_file", "final_answer"]
    action_input: str
    confidence: float  # Fail fast if < 0.5

# 3. Tool execution with retry + timeout
async def execute_tool_with_timeout(tool_fn, args, timeout_s=10, max_retries=2):
    ...

# 4. State persistence (survive API failures)
class AgentCheckpoint:
    def save(self, state: dict, step: int): ...
    def load(self, run_id: str) -> dict: ...

Evaluation protocol:

# Run 100 independent trials with varied inputs
# Record:
results = {
    "success": 0,          # Task fully completed correctly
    "partial": 0,          # Task attempted, incomplete
    "failed_gracefully": 0, # Hit budget limit, returned partial result
    "crashed": 0,          # Unhandled exception
    "cost_per_run": [],    # USD
    "steps_per_run": [],   # Step count
    "tokens_per_run": [],  # Token count
    "wall_time_per_run": [] # Seconds
}

# Compute:
success_rate = results["success"] / 100
mean_cost = sum(results["cost_per_run"]) / 100
p99_latency = sorted(results["wall_time_per_run"])[99]

Acceptance Criteria

  • Success rate ≥ 80% over 100 runs (with defined, verifiable success criterion)

  • Zero unhandled crashes (all failures are graceful — either budget exhausted or explicit error return)

  • Mean cost per run ≤ $0.05 (or equivalent local inference cost)

  • p99 wall-clock time ≤ 120 seconds

  • Circuit breaker triggers and recovers correctly in simulated API failure test

  • Budget enforcer prevents runaway cost: inject an infinite loop; agent must halt within 15 steps

  • Structured output parsing: 100% of LLM calls produce valid structured output (0 parse failures)

  • Evaluation methodology documented: how success/failure is determined, who the judge is

Portfolio Signal

The 80% success rate criterion over 100 runs is the differentiator. Anyone can build an agent that works once. Measuring reliability statistically, with a reproducible evaluation protocol, demonstrates production engineering thinking. The zero-crash criterion and budget protection are reliability signals.

Portfolio Mapping

Project

Primary Signal

Secondary Signal

Time Estimate

QLoRA Fine-Tune

Training engineering, evaluation rigor

Resource efficiency

15-25 hours

RAG System

Architecture depth, measurement discipline

Optimization methodology

20-30 hours

Reliable Agent

Production reliability, systems thinking

Cost efficiency

20-30 hours

Documentation requirement for all three: Each project needs a README.md in its own directory with:

  1. Problem statement (2-3 sentences)

  2. Architecture diagram or system description

  3. Results table with quantitative scores

  4. Failure modes encountered and how they were handled

  5. What you’d do differently with more time/resources

  6. Reproduction instructions (exact commands, package versions)

The fifth point — what you’d do differently — is often what separates a junior portfolio from a senior one. It demonstrates honest calibration.

Return to README.md · Previous: 05_multimodal_and_frontier_models.md · Continue to Phase 5: ../06_phase_5_production_and_mlops/README.md