Rung 4: Transformer Lab

Month 8 | Phase 3: Modern Architectures | ~35–45 hours total

Transformers are the dominant architecture of modern ML. This rung proves you can navigate the transformer ecosystem — not just use it, but understand attention mechanisms, implement or substantially modify a transformer, publish a model card, and communicate your findings to a public audience. The combination of code + model card + blog post is the minimum viable proof for this rung, because technical work without communication is a private exercise, not a portfolio artifact.

Scope Options (Choose One)

What Must Be Implemented (Path A)

For the from-scratch transformer path, you must implement and be able to explain:

class MultiHeadSelfAttention(nn.Module):
    # You implement this from the attention formula:
    # Attention(Q, K, V) = softmax(QK^T / sqrt(d_k)) V
    # Must implement: query/key/value projections, scaled dot-product,
    # causal mask, multi-head split and concat, output projection

class TransformerBlock(nn.Module):
    # Pre-norm architecture (norm before attention, not after)
    # Must explain: why pre-norm is more stable than post-norm in practice

class GPTLanguageModel(nn.Module):
    # Token embedding + positional embedding + stack of blocks + output head

Every class must have a docstring explaining what it does architecturally, and every non-trivial computation (the sqrt(d_k) scaling, the causal mask implementation, the positional embedding indexing) must have an inline comment explaining the why.

HuggingFace Model Card Requirements

The model card on HuggingFace Hub (not optional — publish it) must contain:

---
language: [en]  # or your target language
license: mit
tags:
  - text-generation  # or your task
  - transformers
datasets:
  - [your corpus name or link]
metrics:
  - [perplexity / accuracy / F1 — whichever is appropriate]
---

# [Model Name]

## Model Description
[2–3 sentences: what it does, what architecture, what it was trained on]

## Training Data
[Exact dataset, how it was preprocessed, size in tokens/samples]

## Training Procedure
[Hardware used, training time, optimizer, learning rate schedule, batch size, 
 number of epochs/steps]

## Evaluation Results
| Metric | Value |
|--------|-------|
| [Primary metric] | [Value] |
| [Baseline (random/majority class)] | [Value] |

## Intended Use
[What this model is for and what it is not for]

## Limitations
[At least 3 honest limitations: data coverage, generalization, known failure modes]

## Sample Outputs
[2–3 example inputs and outputs]

## Technical Notes
[Any interesting findings from training: convergence behavior, what hyperparameters 
 mattered most, what failed]

This model card standard is important: it forces you to think about what your model actually does and doesn’t do, which is a production-thinking discipline.

The Blog Post: Minimum Standard

This is where most people fail this rung. The blog post cannot be:

  • A step-by-step tutorial you followed (“First, I installed transformers. Then I ran trainer.train()”)

  • A results summary without analysis (“I achieved 87% accuracy”)

  • A rephrasing of the paper abstract

The blog post must contain three specific sections that prove original engagement:

Section 1: Something I Learned Something non-obvious you discovered by actually doing this. Not “transformers are good at NLP.” Something specific: “When I increased context length from 256 to 512 tokens, loss decreased but training time increased 3.8×, not 2× — here’s why the quadratic attention scaling caused this and what it means for scaling decisions…”

Section 2: Something That Surprised Me A finding that contradicted your prior expectation. “I expected larger batch size to consistently improve training stability, but on my dataset, batch size above 64 caused…”

Section 3: Something That Failed An experiment you ran that didn’t work, why you think it failed, and what you would do differently. This section is the highest-signal part of the post for a sophisticated reader. It proves you ran your own experiments rather than following a script.

Length: 800–1500 words. Quality over length. A 1000-word post with the three sections above is worth more than a 3000-word tutorial transcript.

Where to publish: Substack, Medium (specifically Towards Data Science for discoverability), or your personal site. Link from your GitHub README. Announce on LinkedIn. Measure engagement: a meaningful signal is ≥ 3 thoughtful comments or responses from ML practitioners (not bots, not “great post!”).

Acceptance Criteria

  • Code published on GitHub with README including: architecture description, training setup, sample outputs, link to model card

  • Model card published on HuggingFace Hub meeting the template above

  • Blog post published on a public platform meeting the three-section standard

  • Training run is documented: final loss/accuracy, training curve image in README, hardware used and approximate training time

  • For Path A: model generates coherent completions (visible in README with examples)

  • For Path B: fine-tuned model outperforms zero-shot baseline by ≥ 10 percentage points absolute on the primary metric; domain-specific task is documented with justification for why it matters

  • At least one ablation or experiment is documented: what happened when you changed one meaningful hyperparameter (context length, number of heads, learning rate, architecture variant)

  • GitHub README links to: model card, blog post, training notebook (if applicable)

  • ≥ 12 commits in git history showing development progression

Time Estimate

Task

Estimated Hours

Reading reference implementations + understanding architecture

5–7

Core implementation (Path A) or fine-tuning setup (Path B)

8–12

Training runs + debugging

6–8

Ablation experiments

3–4

Model card writing

2–3

Blog post drafting + editing

4–5

README + GitHub cleanup

2–3

Total

30–42 hours

What Weakens This Rung

  • Using torch.nn.MultiheadAttention as a complete black box (Path A): if you can’t explain what’s inside it, use it as a reference but implement your own. The point is understanding the mechanism.

  • Fine-tuning on a benchmark dataset (Path B): SST-2, IMDB, MNLI, SQuAD — every HuggingFace tutorial uses these. They signal you followed a tutorial. The domain-specific constraint is not optional decoration.

  • No ablation or experiment: a model that was trained once with the tutorial hyperparameters and submitted is not experimental evidence of understanding. Run one meaningful variation.

  • Model card with empty limitations section: “None” or “N/A” in the limitations field signals that you have not thought about when your model fails. Every model has limitations. Stating them is intellectual honesty; omitting them is a red flag.

  • Blog post that is a tutorial: if your blog post could be reproduced by someone who never touched the data or the code, it is not your blog post — it is a paraphrase of the tutorial. The three specific sections (learned, surprised, failed) exist precisely to prevent this.

  • Broken model card link: publish the model to HuggingFace Hub and verify the link works before including it in your README. A 404 is worse than no link.

  • Training on GPU you don’t explain: if the README says “trained on A100 for 3 hours” but you have no way to run this on a Colab free tier or Kaggle, include instructions for a smaller configuration that is runnable on available hardware.

Return to README.md · Previous: 03_rung_3_deep_learning_from_scratch.md · Next: 05_rung_5_llm_engineering.md