05 — Hardware & Local Setup

The hardware question for an ML learner is simpler than the internet makes it sound: you need a decent CPU, enough RAM to hold a dataset in memory, an SSD so data loading isn’t the bottleneck, and — depending on the phase — either a capable Apple Silicon chip or access to cloud GPU. You do not need to buy an RTX 4090. Not in month 1, possibly not ever, depending on your goals.

This document gives you the honest assessment of what works, what’s a waste of money, and how to get the most out of hardware you likely already have.

What You Actually Need Locally

The Minimum Viable ML Setup (2025)

| Component | Minimum | Recommended | Why |\n|———–|———|————-|—–|\n| RAM | 16GB | 32GB | Training on local datasets, running inference on 7B models, multiple processes |\n| Storage | 256GB SSD | 512GB NVMe SSD | Datasets, model weights, conda envs eat space fast |\n| CPU | Any modern 8-core | M2 Pro / M3 Pro or better | Data preprocessing, fast iteration on CPU |\n| GPU | Not required locally | Apple MPS (built-in) or none | Cloud handles GPU workloads |\n| Display | Any | External monitor | Two screens: code + documentation |\n| Internet | Stable broadband | Stable broadband (50Mbps+) | Downloading datasets, cloud GPU access |\n\n### The RAM Question

16GB is survivable. 32GB is where you stop thinking about RAM and start thinking about ML.

What eats RAM in ML work:

  • Python + Jupyter base: ~1-2GB

  • Loading a medium dataset (CIFAR-100 in memory): ~500MB

  • A transformer model loaded in float32 (e.g., BERT-base): ~440MB

  • Running multiple experiments in parallel: 2-4x multiplier

  • Loading a 7B LLM for inference (in 4-bit quantization): ~4-6GB RAM + 4-6GB GPU memory

The 16GB threshold: You can do everything in months 1-8 on 16GB. You will start hitting limits at month 9-10 when running larger models locally for inference or when your datasets grow. If you’re on 8GB, you’ll hit walls earlier — plan on cloud earlier.

M-series Apple Silicon unified memory: Unified memory is shared between CPU and MPS (GPU). A 16GB M2 means both your Python process and the MPS GPU share that 16GB. Effective available GPU memory for a model: ~10-12GB (the rest is OS + process overhead). This is still excellent for learning purposes.

Apple Silicon (M1/M2/M3) — The Full Picture

If you’re on an Apple Silicon Mac, you have a capable ML machine that doesn’t require any cloud GPU for months 1-8.

M-Series Chip Hierarchy for ML (2025)

Chip

Unified Memory Options

MPS Performance

Best For

M1

8GB, 16GB

Baseline

Months 1-6. 8GB is tight.

M1 Pro/Max

16GB, 32GB, 64GB

2x M1

Months 1-9

M2

8GB, 16GB, 24GB

~15% over M1

Months 1-7

M2 Pro/Max

16GB, 32GB, 96GB

Strong

Months 1-10, 7B inference

M3

8GB, 16GB, 24GB

~20% over M2

Months 1-8

M3 Pro/Max

18GB, 36GB, 128GB

Very strong

Months 1-12, some fine-tuning

For serious ML work on Apple Silicon:

  • M2 Pro with 16GB: Solid through months 1-8

  • M2 Max with 32GB: Can fine-tune 7B models with LoRA; sufficient through most of the 13-month plan

  • M3 Max with 64GB+: Genuinely capable for fine-tuning 13B models locally

PyTorch MPS — What Works in 2025

PyTorch 2.4.x MPS backend status:

# Check MPS availability
import torch
print(torch.backends.mps.is_available())  # True on M1/M2/M3
print(torch.backends.mps.is_built())       # True if compiled with MPS support

# Use MPS
device = torch.device("mps")
model = model.to(device)
tensor = tensor.to(device)

Operations that work well on MPS:

  • All standard convolution operations ✅

  • Matrix multiplication (linear layers) ✅

  • Attention (standard PyTorch attention) ✅

  • BatchNorm, LayerNorm ✅

  • Most loss functions ✅

  • Adam, SGD, AdamW optimizers ✅

  • Automatic mixed precision (float16) ✅

Known MPS limitations (as of PyTorch 2.4.x):

  • FlashAttention: Not supported on MPS. Use standard attention or torch.nn.functional.scaled_dot_product_attention (slower but works). ⚠️

  • Some sparse operations: Fall back to CPU. Performance varies. ⚠️

  • bfloat16: Limited support. Use float16 instead for mixed precision. ⚠️

  • Certain custom CUDA kernels: Community libraries (xformers, deepspeed) have partial or no MPS support. ❌

  • Multi-device training: Only one MPS device per machine. No MPS equivalent of DataParallel. ❌

Fallback behavior: When an operation isn’t supported on MPS, PyTorch falls back to CPU silently. This means your code won’t crash, but it will be slower than expected. You can detect fallbacks:

# Enable MPS fallback (on by default, but explicit is better)
import os
os.environ["PYTORCH_ENABLE_MPS_FALLBACK"] = "1"

# To debug which ops are falling back to CPU:
os.environ["PYTORCH_MPS_HIGH_WATERMARK_RATIO"] = "0.0"  # Force OOM instead of fallback

Practical MPS Performance Benchmarks

Rough real-world numbers on an M2 Pro (16GB):

Task

CPU (M2 Pro)

MPS (M2 Pro)

A100 (cloud)

ResNet50, CIFAR-10, 1 epoch

~45s

~12s

~2s

BERT fine-tune, GLUE/MRPC, 1 epoch

~180s

~45s

~8s

GPT-2 small forward pass, batch=16

~0.8s

~0.15s

~0.02s

7B model inference (4-bit), 1 token

~0.3s

~0.05s

~0.01s

The verdict: MPS is 3-5x faster than CPU for typical DL workloads. Not competitive with cloud A100 for serious training, but entirely sufficient for: learning, prototyping, running experiments with small/medium datasets, and local inference on models up to 7B parameters.

Storage: Why an SSD Matters

Data loading is often the training bottleneck. A slow hard drive (HDD) can mean your GPU sits idle 30-40% of the time waiting for the next batch. On macOS, you almost certainly have an NVMe SSD already — verify with:

# Check disk speed
diskutil info disk0 | grep "Medium Type"
# Output: Medium Type: Solid State

# Benchmark read speed (rough)
dd if=/dev/zero of=/tmp/testfile bs=1m count=4096
# ~2000-7000 MB/s is normal for M-series internal SSD

Storage consumption reality check over 13 months:

Item

Space

conda/venv environments (3-5 active)

5-15GB

Downloaded datasets (CIFAR, ImageNet subset, HF datasets)

20-100GB

Model checkpoints (training experiments)

10-50GB

Python packages (pip/conda cache)

3-10GB

Realistic total

40-175GB

Recommendation: 512GB storage is the practical minimum for a 13-month ML journey where you’re keeping datasets locally. 1TB is comfortable. If you’re on 256GB, be aggressive about: removing unused conda environments (conda env remove -n env_name), clearing pip cache (pip cache purge), and using cloud storage (Google Drive / S3) for large datasets rather than keeping them local.

External GPU (eGPU) — Honest Assessment

The short answer: eGPU is not worth it for Apple Silicon Macs in 2025.

Here’s why:

  1. Thunderbolt bandwidth limitation. Thunderbolt 4 / USB4 provides ~40Gbps. PCIe 4.0 x16 (what a GPU needs for full performance) is ~256Gbps. An eGPU via Thunderbolt runs at ~PCIe x4 speeds — roughly 25% of the GPU’s theoretical bandwidth. Training throughput is significantly below what the card would deliver in a desktop.

  2. CUDA doesn’t run on macOS. External NVIDIA GPUs are not supported on macOS. Apple removed CUDA support from macOS in 2019. An eGPU on a Mac would have to use AMD or Apple Silicon chip — the former has poor ML support, the latter doesn’t exist as a separate card.

  3. Windows eGPU + CUDA works but is complicated. If you have a Windows laptop with Thunderbolt, an eGPU with an RTX card can work for CUDA workloads. But the bandwidth penalty is real (~25-40% slower than native PCIe), driver issues are common, and the setup is high-friction.

  4. Cost-to-performance. An eGPU enclosure costs $150-300. An RTX 4090 for the enclosure costs $1,800-2,200. Total: ~$2,000 for a setup that runs at 60-70% of a desktop 4090’s performance. Versus: $2,000 on RunPod GPU-hours = ~1,100 A100 hours. The cloud math wins unless you’re running the same type of workload continuously for 2+ years.

The only reasonable eGPU scenario: You already have an NVIDIA GPU from an old desktop, you have a compatible Windows laptop, and you want to avoid cloud costs for a specific period. Even then, verify Thunderbolt compatibility before buying the enclosure.

Buying Recommendations (If Upgrading)

If you’re making a hardware decision for the 13-month roadmap:

Don’t buy anything new for months 1-6. Whatever Mac you have is sufficient. Cloud GPU handles the rest.

If you’re buying a new Mac (within 2025-2026):

  • Minimum: M3 Pro, 18GB unified memory — handles months 1-10 comfortably

  • Recommended: M3 Max, 36GB — handles the full 13 months including local 7B model work

  • Skip the base M3 with 8GB — you’ll hit RAM limits by month 5

If you’re on a budget and need to upgrade:

  • Prioritize RAM over CPU speed. 32GB > 16GB matters more than M3 vs M2 for ML workloads.

  • Don’t buy more local storage if cloud storage solves the dataset problem. HuggingFace Hub + Kaggle datasets + Google Drive handle data at zero cost.

When to consider a dedicated GPU workstation:

  • You’re doing this full-time or near-full-time (not 10-15 hrs/week alongside a job)

  • You have specific workloads that require multi-GPU training

  • Your cloud compute costs exceed $150-200/month consistently

  • You have a specific research need that requires low-latency local GPU access

For a part-time learner on a 13-month structured plan: cloud compute is almost always the economically correct answer. The breakeven on a $2,000 RTX 4090 desktop vs. cloud GPU rental at $200/month is 10 months — and that’s assuming 100% utilization, which a learner doesn’t have.

Quick Hardware Checklist

Before starting the 13-month plan:

  • Python 3.11 installed and working (see 01_python_environment_setup.md)

  • torch.backends.mps.is_available() returns True (Apple Silicon)

  • At least 50GB free disk space

  • Kaggle account created (kaggle.com) — free GPU

  • W&B account created (wandb.ai) — free experiment tracking

  • Google Colab accessible (colab.research.google.com)

  • HuggingFace account created (huggingface.co) — model hub

  • VS Code installed with Jupyter + Pylance + Ruff extensions

If all boxes are checked: your local environment is ready. Cloud handles the rest.

Return to README.md · Previous: 04_version_control_for_ml.md