The Lab Notebook — Predict, Then Measure

The difference between an ML engineer and an ML scientist is that the scientist writes down what they expect to happen before they run the experiment. After 13 months of doing this, you will have built something most practitioners don’t have: a verified, calibrated intuition.

Why This Is the Single Most Powerful Habit

Most people who study ML run experiments like homework. They execute the code, look at the output, and nod or frown. They don’t write down what they expected to see beforehand. This means they can’t tell the difference between:

  • “I understand this and my prediction was correct” (genuine knowledge)

  • “I got a good result but for the wrong reasons” (false confidence)

  • “I got a bad result and I don’t know why” (confusion, diagnosed)

  • “I got a bad result but I can name exactly why” (actual learning)

The lab notebook forces you to have a model of the world before you run the experiment. When the results contradict your prediction, that contradiction is the learning. When the results confirm your prediction, that confirmation is the evidence that your mental model is accurate.

Over 13 months and ~200 lab notebook entries, you will build a calibrated, evidence-based intuition for how ML systems behave. This is what separates practitioners who can diagnose problems from practitioners who can only run tutorials.

What Most ML Students Get Wrong

“Treating experiments as homework instead of hypothesis tests.”

Homework has a correct answer at the back of the textbook. You run the code, check that the loss goes down, mark it complete, move on.

Hypothesis tests have a prediction, a result, and a reconciliation. The reconciliation is the work. “I predicted the learning rate of 0.01 would converge faster than 0.001 because the gradient landscape is smooth in the early layers. It didn’t. The validation loss spiked at epoch 3. Updated belief: the gradient landscape is not smooth here, or the batch size is too small for stable estimates at that learning rate. Next experiment: fix the learning rate and vary batch size.”

That’s a paragraph that took 3 minutes to write. It is worth more than 3 hours of running code without predictions.

The Lab Notebook Format

Every entry uses this structure. Fill in what’s relevant; skip what isn’t. The point is the hypothesis.

LAB NOTEBOOK ENTRY
==================
Date: YYYY-MM-DD
Sprint #: ____
Topic: ____

HYPOTHESIS
----------
I predict [X] will happen because [Y].

(Be specific. "Loss will decrease" is not a hypothesis. 
"Validation loss will plateau after epoch 15 because the 
model capacity is too small relative to dataset complexity" 
is a hypothesis.)

EXPERIMENT SETUP
----------------
- What I'm testing: 
- Dataset / input:
- Model / method:
- Key hyperparameters:
- Fixed variables (what I'm NOT changing):
- How I'll measure outcome:

RESULTS
-------
- What actually happened:
- Quantitative results (numbers, not vibes):
- Plots generated (list them with filenames):

WHAT WAS SURPRISING
-------------------
(If nothing was surprising, you already knew this. 
That means you didn't push the boundary of your understanding.
Adjust the next experiment to be harder.)

UPDATED MENTAL MODEL
--------------------
Before this experiment, I believed: ____
After this experiment, I now believe: ____
Confidence in updated belief: Low / Medium / High
Next question this raises: ____

Daily Entry Template (5 Minutes Max)

For days when you’re doing the daily 30-minute practice rather than a full sprint session:

DAILY NOTE — [DATE]
===================
What I read/watched/coded today: 

One thing I understood that I didn't before:

One question I can't answer yet:

Tomorrow I want to:

This is not a journal. It is a breadcrumb trail. In Month 8, you will read your Month 2 daily notes and see exactly how your thinking evolved. That is worth more than any textbook.

Weekly Review Template (15 Minutes)

Run this on Friday as part of the daily practice rotation.

WEEKLY REVIEW — Week of [DATE]
================================
ENTRIES THIS WEEK: ____
(If < 3: what happened? Log the reason.)

BEST PREDICTION THIS WEEK:
(The one you got right and understood why.)

WORST PREDICTION THIS WEEK:
(The one you got wrong. This is the most important entry.)

PATTERN I'M NOTICING:
(Is there a category of predictions I keep getting wrong? 
E.g., "I consistently underestimate training time on larger datasets."
This is an actual mental model gap. Fix it.)

MENTAL MODEL UPDATES THIS WEEK:
(List 1-3 beliefs you updated based on evidence.)

AM I BUILDING INTUITION OR JUST COMPLETING TASKS?
[ ] Building intuition — predictions are getting more accurate
[ ] Completing tasks — executing without genuine hypothesis formation
[ ] Not sure — will run a harder experiment next week to test

Where to Keep the Lab Notebook

Recommended: Obsidian

Why Obsidian and not Notion, Google Docs, or paper:

  1. Local-first: Your notes don’t disappear if a service shuts down or you lose internet access

  2. Plain Markdown: Every entry is a .md file you can grep, version-control, and export

  3. Bidirectional links: You can link [[learning_rate_behavior]] across entries and build a knowledge graph of your own experiments

  4. Graph view: Visually see which concepts you’ve explored and which remain isolated

  5. Free: No subscription, no lock-in

Create a vault structure:

obsidian_vault/
├── lab_notebook/
│   ├── sprint_01/
│   ├── sprint_02/
│   └── ...
├── mental_models/
│   ├── optimization.md
│   ├── regularization.md
│   └── ...
├── paper_notes/
└── daily_notes/

Sync it to a private GitHub repo. Push weekly. This is your intellectual property — back it up.

How to Build Intuition Over Time

The lab notebook is not useful on any single day. It becomes useful when you read it longitudinally.

Month 3 protocol: Read your Month 1 entries. Identify 3 beliefs you held that were wrong.

Month 7 protocol: Read your Month 3 entries. Identify where your mental models were incomplete.

Month 13 protocol: Read the entire notebook. Map the evolution of your understanding from first principles to frontier. This is the evidence you need for the portfolio pitch.

The lab notebook is the proof that you didn’t just consume information — you built knowledge through empirical iteration. That is what a PhD student does. That is what you are doing.