04 — Version Control for ML

ML projects have unique version control challenges that regular software development doesn’t face: large binary files (models, datasets), non-deterministic experiments, notebook outputs that bloat history, and the need to reproduce a specific result from 3 months ago. Standard git practices are necessary but not sufficient. This document covers what you need on top of them.

The short version: use git for code, DVC for data and models, and W&B for experiment results. Never commit binary model weights to git.

---

Git Fundamentals for ML (The Non-Obvious Parts)

The ML .gitignore

Every ML repo needs this .gitignore. The defaults that everyone forgets:

# Python
__pycache__/
*.py[cod]
*$py.class
*.so
.Python
.env
.venv/
venv/
env/
*.egg-info/
dist/
build/
.eggs/

# Jupyter Notebooks — outputs stripped by nbstripout, but still exclude checkpoints
.ipynb_checkpoints/
*.ipynb_checkpoints

# Model weights and checkpoints — NEVER commit these to git
*.pt
*.pth
*.ckpt
*.pkl
*.pickle
*.h5
*.hdf5
*.safetensors
checkpoints/
saved_models/
weights/

# Data — NEVER commit raw or processed data to git
data/raw/
data/processed/
data/interim/
data/external/
*.csv
*.parquet
*.feather
*.arrow
*.json.gz
*.jsonl.gz

# Exceptions: small config files that happen to be .json ARE okay
!config/*.json
!configs/*.yaml

# Logs and experiment outputs
logs/
runs/
wandb/
mlruns/
outputs/
results/
*.log

# IDE
.vscode/settings.json
.idea/
*.swp
*.swo

# OS
.DS_Store
.DS_Store?
._*
Thumbs.db

# Environment
.env
.envrc
*.env.local

# Large files
*.zip
*.tar.gz
*.tar.bz2

What most people get wrong: Committing *.pt model files. A fine-tuned BERT model is 440MB. After 10 experiments, your repo is 4GB. GitHub will reject pushes over 100MB. Even if it doesn’t reject, your git clone time goes from 3 seconds to 40 minutes. Models go in DVC, S3, or HuggingFace Hub — not in git.

---

Git Workflow for ML Projects

Branch strategy for ML experiments

# Main branch: clean, reproducible, documented experiments
main

# Feature branches: one per experiment or model variant
experiment/resnet50-baseline
experiment/resnet50-larger-batch
experiment/efficientnet-comparison
feature/data-augmentation
feature/lr-scheduler

# Don't branch on every hyperparameter change
# That's what W&B is for

Commit messages for ML projects

Standard commit message format for ML work:

# Structure: <type>(<scope>): <description>
# Types: feat, fix, exp, data, model, refactor, docs

git commit -m "feat(training): add cosine LR scheduler with warmup"
git commit -m "exp(baseline): train ResNet50 on CIFAR-10, val_acc=91.2%"
git commit -m "fix(data): correct label encoding for class 7"
git commit -m "model(architecture): add dropout layers to classifier head"
git commit -m "data: add preprocessing pipeline for ImageNet normalization"

Log results in commits, not just descriptions:

git commit -m "exp(transformer): 3-layer GPT on Shakespeare

- val_loss: 1.47 after 5000 iterations
- train_loss: 1.21
- params: 10.7M
- training time: 12min on A100
- full results: wandb.ai/user/project/runs/abc123"

Tagging reproducible baselines

# Tag the commit that produced your best result
git tag -a v0.1-baseline -m "ResNet50 baseline: val_acc=91.2%, 50 epochs, lr=0.01"
git push origin v0.1-baseline

# Later, to reproduce:
git checkout v0.1-baseline
# Restore data: dvc checkout (see DVC section)
# Run: python train.py --config configs/resnet50_baseline.yaml

---

DVC — Data Version Control

DVC (Data Version Control) is the tool for versioning datasets and model files. It works like git but for large binary files — it stores the actual files in remote storage (S3, GCS, Azure, SSH, or local) and tracks only small metadata files (.dvc) in git.

Install DVC

pip install dvc

# With specific storage backend support
pip install dvc[s3]        # For AWS S3
pip install dvc[gs]        # For Google Cloud Storage
pip install dvc[gdrive]    # For Google Drive (easiest for learners)

Initialize DVC in your project

# In your git repo
dvc init

# This creates:
# .dvc/             (DVC configuration)
# .dvcignore        (like .gitignore for DVC)

# Commit the DVC initialization
git add .dvc/ .dvcignore
git commit -m "chore: initialize DVC"

Track data files with DVC

# Add a dataset to DVC tracking
dvc add data/raw/imagenet_subset.tar.gz

# This creates:
# data/raw/imagenet_subset.tar.gz.dvc  (small metadata file, committed to git)
# data/raw/.gitignore                  (DVC adds the actual file to .gitignore)

# Commit the .dvc file to git (not the data)
git add data/raw/imagenet_subset.tar.gz.dvc data/raw/.gitignore
git commit -m "data: add ImageNet subset (DVC tracked)"

# Track model weights
dvc add models/resnet50_epoch50.pt
git add models/resnet50_epoch50.pt.dvc models/.gitignore
git commit -m "model: add ResNet50 baseline checkpoint (DVC tracked)"

Set up remote storage (Google Drive — easiest for learners)

# Google Drive remote (free, no setup required)
dvc remote add -d gdrive gdrive://FOLDER_ID_FROM_DRIVE_URL

# Authenticate (opens browser)
dvc push  # First push triggers auth

# Push data to remote
dvc push

# Pull data on a new machine
git clone <repo_url>
dvc pull  # Downloads all DVC-tracked files from remote

DVC Pipelines (reproducible ML pipelines)

For production-grade reproducibility, define your pipeline in dvc.yaml:

stages:
  preprocess:
    cmd: python src/data/preprocess.py --input data/raw --output data/processed
    deps:
      - src/data/preprocess.py
      - data/raw
    outs:
      - data/processed

  train:
    cmd: python train.py --config configs/baseline.yaml
    deps:
      - train.py
      - src/models/resnet.py
      - data/processed
      - configs/baseline.yaml
    outs:
      - models/checkpoint_best.pt
    metrics:
      - metrics/train_metrics.json:
          cache: false

  evaluate:
    cmd: python evaluate.py --checkpoint models/checkpoint_best.pt
    deps:
      - evaluate.py
      - models/checkpoint_best.pt
      - data/processed
    metrics:
      - metrics/eval_metrics.json:
          cache: false

Run the pipeline:

dvc repro   # Run changed stages only (like make, but smart)
dvc dag     # Visualize the pipeline DAG

---

The Canonical ML Repository Structure

Based on community consensus (Cookiecutter Data Science + modern additions):

my_ml_project/
│
├── README.md                   # Project overview, results, reproduction steps
├── pyproject.toml              # Python dependencies
├── uv.lock                     # Pinned dependencies
├── .gitignore                  # ML-specific ignores (see above)
├── .pre-commit-config.yaml     # Pre-commit hooks
├── dvc.yaml                    # DVC pipeline definition
├── dvc.lock                    # DVC pipeline lock file (commit to git)
│
├── configs/                    # Experiment configurations (YAML)
│   ├── baseline.yaml
│   └── ablation_lr.yaml
│
├── data/                       # All data (mostly DVC-tracked, not in git)
│   ├── raw/                    # Original, immutable data
│   ├── processed/              # Cleaned, feature-engineered data
│   └── external/               # Third-party data
│
├── models/                     # Trained model checkpoints (DVC-tracked)
│
├── notebooks/                  # Exploration notebooks (nbstripout active)
│   ├── 01_eda.ipynb
│   ├── 02_baseline_experiments.ipynb
│   └── 03_error_analysis.ipynb
│
├── src/                        # Source code (importable package)
│   ├── __init__.py
│   ├── data/                   # Data loading and processing
│   │   ├── __init__.py
│   │   ├── dataset.py
│   │   └── transforms.py
│   ├── models/                 # Model definitions
│   │   ├── __init__.py
│   │   └── resnet.py
│   └── utils/                  # Utilities
│       ├── __init__.py
│       ├── metrics.py
│       └── visualization.py
│
├── train.py                    # Training entry point
├── evaluate.py                 # Evaluation entry point
├── predict.py                  # Inference entry point
│
├── tests/                      # Unit tests
│   ├── test_dataset.py
│   └── test_model.py
│
├── reports/                    # Analysis reports, figures
│   └── figures/
│
└── logs/                       # Training logs (gitignored)

Why src/ layout?

The src/ layout (putting your package under src/) prevents the common Python import bug where import mypackage resolves to the local directory instead of the installed package. It forces proper installation and prevents namespace collisions.

# With src/ layout, install your own package in development mode:
pip install -e .
# or
uv pip install -e .

# Now you can do this from anywhere:
from src.models.resnet import ResNet50

---

Experiment Configuration: YAML + Hydra/OmegaConf

Hardcoding hyperparameters in your training script is how you lose track of what configuration produced what result. Use configuration files:

# configs/baseline.yaml
model:
  architecture: resnet50
  pretrained: true
  num_classes: 10
  dropout: 0.3

training:
  epochs: 50
  batch_size: 32
  learning_rate: 0.001
  optimizer: adam
  scheduler: cosine
  warmup_epochs: 5

data:
  dataset: cifar10
  train_split: 0.8
  augmentation: standard
  num_workers: 4

logging:
  project: ml-project
  experiment_name: resnet50-baseline

Load in your training script:

import yaml

with open("configs/baseline.yaml") as f:
    config = yaml.safe_load(f)

# Access
lr = config["training"]["learning_rate"]

For larger projects, use Hydra for config composition and CLI overrides:

# Override any config value from command line:
python train.py training.learning_rate=0.01 training.batch_size=64

---

What Most ML Engineers Do Wrong with Version Control

1. Not tracking experiment configs. Running 20 experiments but not saving which config produced which result. The fix: commit the YAML config file alongside the .dvc model checkpoint, and log the config to W&B.

2. Committing data to git. The data folder is not source code. It changes independently, it’s binary, and it’s large. DVC exists precisely for this. Set up DVC before you have 50GB of data already in your git history.

3. One giant notebook, never refactored. The final_final_v2_ACTUAL_FINAL.ipynb problem. Use numbered notebooks for exploration phases, refactor logic to .py files before it becomes unmaintainable.

4. No tests. Even a minimal tests/test_dataset.py that verifies your Dataset class returns the right shapes saves hours of debugging mysterious shape errors during training. Write at least data loading and model forward pass tests.

---

Return to README.md · Previous: 03_development_environment.md · Next: 05_hardware_and_local_setup.md