05 — Datasets Canon¶
Every ML engineer has a list of models they’ve trained. The ones worth talking to have a list of datasets they’ve understood — where the data came from, what its biases are, what it measures and what it doesn’t. Benchmark performance on a dataset you don’t understand is theater. The datasets below are not just training data — they’re the shared vocabulary of the field.
What most people get wrong: They download MNIST, hit 99% accuracy, and feel competent. MNIST is a solved toy. Its purpose is to verify your training loop works, not to measure whether you understand deep learning. Move through it in a day and never look back.
How to Work With a Dataset (Not Just Download It)¶
Understand provenance: Where was this data collected? Who labeled it? What biases were introduced?
Visualize samples: Look at 100 random examples before writing any model code.
Know the evaluation metric: Accuracy on MNIST means something different from F1 on SQuAD.
Understand train/val/test splits: Don’t evaluate on the training set. Don’t tune on the test set.
Know the SOTA: What’s the current state-of-the-art on this benchmark? Is your model competitive?
Vision Datasets¶
1. MNIST¶
HuggingFace ID: ylecun/mnist
What it is: 70K grayscale 28×28 handwritten digit images (60K train, 10K test). 10 classes (0-9).
Why it’s in the canon: Every deep learning tutorial uses it. Verifying that your training loop, loss function, and optimizer are working correctly.
Current SOTA accuracy: ~99.87% (ensemble methods). A basic CNN should hit 99.2%+.
⚠️ The MNIST Problem: It’s too easy. 99% accuracy doesn’t mean your model is good. It means your code doesn’t crash. Use it as a debugging tool, not a benchmark. LeCun himself has said “If your model can’t get 99% on MNIST, something is wrong.”
Roadmap phase: Month 1-2 (debugging only — move on fast)
2. CIFAR-10 / CIFAR-100¶
HuggingFace ID: uoft-cs/cifar10, uoft-cs/cifar100
What it is: CIFAR-10: 60K 32×32 color images, 10 classes (airplane, automobile, bird, cat, etc.), 6K per class. CIFAR-100: 60K images, 100 fine-grained classes.
Why it’s in the canon: Real complexity, small enough to iterate on quickly. Architectures that work on CIFAR-10 transfer to larger vision problems. Useful for testing new ideas quickly.
Current SOTA: CIFAR-10: ~99.5% (ViT + extensive augmentation). ResNet-56: ~93.03% (the baseline to beat).
Roadmap phase: Months 2-4 (computer vision fundamentals)
3. ImageNet (ILSVRC)¶
HuggingFace ID: ILSVRC/imagenet-1k (requires agreement)
What it is: 1.28M training images, 50K validation, 100K test. 1,000 classes. The benchmark that drove deep learning progress 2012-2020.
Why it’s in the canon: Every major CNN architecture (AlexNet, VGG, ResNet, EfficientNet, ViT) was measured against ImageNet. Understanding ImageNet top-1/top-5 accuracy is table stakes for computer vision discussions.
⚠️ Practical note: ImageNet is 150GB+ and requires registration to download from ImageNet.org. HuggingFace hosts it but requires accepting terms. For learning, use ImageNette (a 10-class subset from fast.ai: frgfm/imagenette) — ~3GB, perfect for local experimentation.
Current SOTA: ~91.1% top-1 (CoAtNet / large ViT models with external pretraining).
Roadmap phase: Month 4-6 (computer vision — use ImageNette locally, ImageNet at compute scale)
4. ImageNette¶
HuggingFace ID: frgfm/imagenette
What it is: 10 easily-classifiable classes from ImageNet (tench, English springer, cassette player, chain saw, church, French horn, garbage truck, gas pump, golf ball, parachute). ~13K images.
Why it’s in the canon: Fast.ai created this specifically for rapid experimentation. Download, train, iterate in an afternoon. The meaningful alternative to CIFAR when you want ImageNet-like images without ImageNet’s cost.
Roadmap phase: Months 3-6 (primary vision experimentation dataset)
NLP Datasets¶
5. GLUE / SuperGLUE¶
HuggingFace ID: nyu-mll/glue, aps/super_glue
What it is: GLUE: Multi-task NLP benchmark. 9 tasks: sentiment analysis, NLI, textual similarity, coreference. SuperGLUE: Harder version with reading comprehension, word sense disambiguation.
Why it’s in the canon: BERT was evaluated on GLUE. Every NLP model reports GLUE/SuperGLUE scores. Reading the GLUE paper explains what “NLU benchmarking” actually measures (and its limitations).
Roadmap phase: Month 6-8 (NLP phase — understand the benchmark before fine-tuning on it)
6. SQuAD 1.1 / SQuAD 2.0¶
HuggingFace ID: rajpurkar/squad, rajpurkar/squad_v2
What it is: Stanford Question Answering Dataset. 100K+ question-answer pairs on Wikipedia articles. SQuAD 2.0 adds unanswerable questions.
Why it’s in the canon: The standard reading comprehension benchmark. Fine-tuning BERT on SQuAD is the canonical NLP fine-tuning tutorial. SQuAD 2.0’s unanswerable questions test a model’s ability to say “I don’t know.”
Roadmap phase: Month 6-8
7. WikiText-103 / WikiText-2¶
HuggingFace ID: Salesforce/wikitext
What it is: Tokens extracted from verified Wikipedia articles. WikiText-103: 103M tokens. WikiText-2: 2M tokens.
Why it’s in the canon: Standard language model benchmarking. Perplexity on WikiText-103 is a common LM evaluation. Small enough that WikiText-2 is usable for training small LMs on a laptop.
Roadmap phase: Month 5-7 (language modeling experiments)
8. Common Crawl¶
HuggingFace: See C4 (allenai/c4), which is a cleaned Common Crawl subset
What it is: Monthly web crawl of ~3 billion web pages, ~250TB uncompressed. The raw data behind most large LLMs.
Why it’s in the canon: Understanding that LLMs are trained on internet text — including its biases, toxicity, and noise — is essential for working with them intelligently. C4 (Colossal Clean Crawled Corpus) is the cleaned version used to train T5 and others.
⚠️ Practical note: Don’t download raw Common Crawl. Use C4 (allenai/c4) via HuggingFace for reasonable experiments.
Roadmap phase: Month 7-9 (LLM pretraining understanding — conceptual, not necessarily a dataset you train on)
9. The Pile¶
HuggingFace ID: EleutherAI/pile (some subsets available; full dataset is ~825GB)
What it is: 825GB diverse English text from 22 sources: books, arXiv papers, GitHub, Wikipedia, DM Mathematics, StackExchange, HackerNews, and more. Created by EleutherAI for training GPT-NeoX.
Why it’s in the canon: The canonical open-source LLM pretraining dataset. Understanding its composition explains a lot about what open-source LLMs know (and don’t know). The Pile’s data card is worth reading in full.
Roadmap phase: Month 8-9 (LLM understanding — read the data card, explore subsets)
10. RedPajama-Data-1T¶
HuggingFace ID: togethercomputer/RedPajama-Data-1T
What it is: 1.2T token open-source replica of the LLaMA training dataset. Includes Common Crawl, C4, GitHub, Books, ArXiv, Wikipedia, StackExchange.
Why it’s in the canon: The open-source reconstruction of what LLaMA was trained on. Essential for anyone wanting to pretrain or continue-pretrain a smaller LLM.
Roadmap phase: Month 8-10 (LLM pretraining)
Instruction Tuning & RLHF Datasets¶
11. Alpaca (Stanford)¶
HuggingFace ID: tatsu-lab/alpaca
What it is: 52K instruction-following examples generated from GPT-3 using self-instruct. The first widely-used instruction tuning dataset for open-source models.
Why it’s in the canon: Alpaca-7B (LLaMA-7B fine-tuned on Alpaca) was the first widely-replicated instruction-following open-source model. Understanding its construction (and limitations — many examples have errors or hallucinations) is essential.
Roadmap phase: Month 9 (instruction tuning phase)
12. Dolly 2.0 (Databricks)¶
HuggingFace ID: databricks/databricks-dolly-15k
What it is: 15K instruction-following examples written entirely by Databricks employees (not AI-generated). The first commercially usable (Apache 2.0 licensed) instruction dataset.
Why it’s in the canon: Licensing matters. Unlike Alpaca (which contains OpenAI outputs and is non-commercial), Dolly 2.0 is fully open. A model fine-tuned only on Dolly can be commercially deployed.
Roadmap phase: Month 9
13. OpenHermes / OpenHermes-2.5¶
HuggingFace ID: teknium/OpenHermes-2.5
What it is: ~1M synthetic instruction-following samples curated from multiple sources. High quality, filtered. Used to train the Hermes series of fine-tuned models.
Why it’s in the canon: Represents the “modern” instruction dataset pipeline: aggregate, filter, deduplicate, quality-check. Compare its scale and composition to Alpaca’s 52K to understand how the field has evolved.
Roadmap phase: Month 9-10
Tabular Datasets¶
14. Adult Income (UCI)¶
HuggingFace ID: scikit-learn/adult-census-income
What it is: 48K rows, census data, binary classification (income >50K or not). Classic fairness benchmark.
Why it’s in the canon: The canonical dataset for studying algorithmic fairness and bias. Gender and race features create predictable biases. Essential for understanding that ML models learn and amplify human biases.
Roadmap phase: Month 3-4 (classical ML + ethics context)
15. Titanic (Kaggle)¶
HuggingFace ID: Available on Kaggle (heptapod/titanic)
What it is: 891 training rows, 418 test rows. Binary classification: survived or not. 11 features.
Why it’s in the canon: The most-used first Kaggle competition dataset. Not because it’s technically interesting (it’s tiny and simple) but because every practitioner has worked through feature engineering on it. It’s the shared “hello world” of tabular ML.
Roadmap phase: Month 1-2 (Kaggle onboarding)
HuggingFace Hub — Datasets Worth Exploring¶
The HuggingFace Datasets Hub hosts 90,000+ datasets as of 2026. These are consistently among the most-downloaded:
Dataset |
HF ID |
Use Case |
|---|---|---|
Common Voice |
|
Speech recognition, audio ML |
MS COCO |
|
Object detection, captioning |
BookCorpus |
|
Language model pretraining |
IMDb |
|
Sentiment classification tutorial |
AG News |
|
Text classification, 4 classes |
HumanEval |
|
Code generation benchmarking |
TruthfulQA |
|
LLM hallucination evaluation |
Dataset Phase Map¶
Phase |
Months |
Primary Datasets |
|---|---|---|
Classical ML |
1-3 |
Titanic, Adult Income, MNIST |
Computer Vision |
3-6 |
CIFAR-10, ImageNette, CIFAR-100 |
NLP Foundations |
4-7 |
SQuAD, GLUE, WikiText-2 |
LLM Pretraining |
7-9 |
WikiText-103, The Pile (subsets), RedPajama |
LLM Fine-tuning |
8-10 |
Alpaca, Dolly, OpenHermes |
Production Systems |
10-13 |
Domain-specific datasets for your use case |
Return to README.md · Previous: 04_tools_and_libraries_canon.md