02 — GPU & Compute Strategy¶

The compute question is the most practically important setup decision you’ll make. You’re likely on a Mac (based on your workspace path), which means no NVIDIA GPU. This document tells you exactly where to run each type of workload, what it costs, and how to avoid paying for GPU time you don’t need yet.

The core principle: most ML learning in months 1-6 can happen without GPU. CPU-speed iteration is actually better for understanding — you’re forced to use small models and small datasets, which means faster iteration loops. GPU is not a prerequisite for learning. It becomes necessary when you hit Transformers (month 5-6 and beyond) and fine-tuning experiments.

The Full Compute Option Matrix¶

Platform	GPU	VRAM	Cost/Month	Cost (INR/month)	Session Limit	Persistent Storage	Best For
Local (Mac M2/M3)	Apple MPS	Unified 16-96GB	$0	₹0	None	✅ Full	Months 1-6, prototyping
Google Colab Free	T4	16GB	$0	₹0	~12hr, unreliable	❌ None	Quick experiments
Google Colab Pro	T4/V100/A100	16-40GB	$11.99	~₹999/month	Longer	✅ Google Drive	Months 5-10, regular training
Kaggle Notebooks	T4 or P100	16GB	$0	₹0	30hr/week	✅ 20GB	Best free option
Lightning AI Free	T4	16GB	$0	₹0	80hr/month	✅ Persistent workspace	Small fine-tuning runs
Vast.ai	RTX 3090+	24GB+	~$5-20+	~₹420-1700	None (on-demand)	Extra cost	Cost-sensitive GPU training
RunPod	RTX 4090/A100	24-80GB	Variable	Variable	None	✅	Reliable cloud GPU
Lambda Labs	A100 80GB	80GB	$1.10/hr	~₹92/hr	None	✅	Production fine-tuning
AWS/GCP/Azure	A100/V100	40-80GB	$20-33/hr	~₹1700-2800/hr	None	✅	Enterprise (avoid unless required)

Platform Deep Dives¶

Your Mac (Apple Silicon MPS) — Use This First¶

If you have an M1, M2, or M3 Mac with unified memory, this is your primary compute resource for months 1-8.

What MPS (Metal Performance Shaders) supports in PyTorch 2.4.x:

Standard tensor operations ✅
Convolutional networks (CNNs) ✅
Transformers / attention ✅
Most autograd operations ✅
Training runs for models up to ~7B parameters (on 32GB+ unified memory) ✅
float16 (half precision) ✅
Some operations fall back to CPU automatically ⚠️

What MPS does NOT support or is slower at:

FlashAttention (not supported as of 2025, use standard attention) ❌
Some sparse operations ❌
Multi-GPU training ❌ (only one MPS device)
bfloat16 has limited support ⚠️

Using MPS in PyTorch:

import torch

# Check availability
device = torch.device("mps" if torch.backends.mps.is_available() else "cpu")
print(f"Using device: {device}")

# Move model and data to MPS
model = MyModel().to(device)
x = torch.randn(batch_size, features).to(device)

# Training loop works identically to CUDA
output = model(x)

Performance reality: MPS on an M2 Pro (16GB) is roughly 3-5x faster than CPU for PyTorch training. Not as fast as an A100, but sufficient for:

CIFAR-10/100 training (full ResNet in minutes)
Fine-tuning small models (DistilBERT, GPT-2 small)
Prototyping any architecture before scaling to cloud GPU
Training word2vec, classical ML models, tabular neural networks

Effective VRAM estimate: With unified memory, a 16GB M2 can effectively use ~10-12GB for PyTorch before performance degrades. A 32GB M2 Max/M3 Max is genuinely capable for 7B parameter inference and some fine-tuning.

Kaggle Notebooks — Best Free GPU (Use This in Months 5-9)¶

This is the correct answer to “free GPU” in 2025. Here’s why Kaggle beats Colab Free:

Feature	Kaggle	Colab Free
GPU type	T4 16GB or P100 16GB	T4 16GB
Weekly GPU hours	30 hours/week guaranteed	Variable, often unavailable at peak
Requires credit card	No	No
Session interruptions	Rare	Common
Persistent storage	20GB	None (session only)
GPU availability	High	Low at peak hours
Outputs saved	✅	❌ (need Google Drive)

How to maximize Kaggle free GPU:

Use GPU sessions only for training runs, not exploration
Save checkpoints to /kaggle/working/ (persists)
Use Kaggle Datasets to store preprocessed data (faster than re-downloading)
Schedule long runs during off-peak hours (late night in US is off-peak)
P100 is better than T4 for float64 scientific computing; T4 is better for float16 deep learning (tensor cores)

Kaggle 30hr/week limit reality check: 30 hours/week × 4 weeks = 120 GPU-hours/month. At a realistic training session usage of 2-4 hours, that’s 30-60 training runs per month. More than sufficient for months 5-9 of learning.

Google Colab Pro — ₹899/month¶

When to upgrade: Month 5-6, when you start training Transformers and Kaggle free tier isn’t enough.

Cost: ₹899/month (India pricing as of 2025)
GPU: T4, V100 (16GB), or A100 40GB (allocation varies; not guaranteed)
Session: Up to 24 hours, no forced disconnections
Storage: Google Drive mount (15GB free, expandable)
Advantage over free: Reliable GPU access, longer sessions, better GPUs when available

Colab Pro+ (₹3,999/month): Not recommended unless you’re running multiple long training jobs. The marginal improvement over Pro is not worth the cost for a learner on a 10-15 hr/week schedule.

Colab tricks worth knowing:

# Check what GPU you got
import subprocess
result = subprocess.run(['nvidia-smi'], capture_output=True, text=True)
print(result.stdout)

# Mount Google Drive (persistent storage)
from google.colab import drive
drive.mount('/content/drive')

# Prevent session timeout (run in a cell)
import time
while True:
    time.sleep(60)
    print("alive")

Lightning AI — 80 Free GPU-hours/month¶

URL: lightning.ai
Free tier: 80 GPU-hours/month (requires phone verification)
GPU: T4 (16GB VRAM)
Advantage: VS Code-like cloud IDE that persists between sessions. Better development experience than Colab notebooks.
Best for: Months 7-10, when you want a persistent cloud dev environment

Vast.ai — Cheapest Cloud GPU (Pay-as-you-go)¶

Vast.ai is a marketplace where GPU owners rent out their machines. It is the cheapest option but with variable reliability.

Current approximate prices (July 2026):

GPU	VRAM	USD/hr	INR/hr (approx)
RTX 3090	24GB	$0.07-0.20	₹6-17
RTX 4090	24GB	$0.25-0.40	₹21-33
A100 40GB	40GB	$0.60-1.00	₹50-83
A100 80GB	80GB	$0.80-1.50	₹67-125
H100 80GB	80GB	$1.80-3.50	₹150-290

Vast.ai usage:

Create account at vast.ai
Filter by GPU model, CUDA version, and reliability score
Choose “on-demand” (reliable) vs “interruptible” (cheaper, can be preempted)
Use the Jupyter or SSH template
Upload code via git clone or file transfer
Always checkpoint frequently (host can reclaim the machine)

What to watch for: Low reliability score hosts (below 95%) interrupt your training more. Filter for >97% reliability. Also verify the CUDA version matches your requirements before renting.

RunPod — More Reliable Than Vast.ai ¶

URL: runpod.io
Pricing: RTX 3090 from ~$0.17/hr, A100 80GB ~$1.64/hr, H100 ~$2.49/hr
Advantage over Vast.ai: Better UX, more reliable machines, persistent storage pods
Best for: Months 10-13, serious fine-tuning experiments where reliability matters

RunPod vs Vast.ai decision: Use Vast.ai for short experiments when you can restart easily. Use RunPod for longer runs (8+ hours) where interruption is costly.

Lambda Labs — Research-Grade Cloud¶

Pricing: A100 80GB ~$1.10-1.29/hr
Advantage: Consistent hardware, academic/researcher-focused, NVIDIA A100s reliably available
Best for: Month 12-13 LLM fine-tuning projects

Phase-by-Phase Compute Recommendation¶

Months	Phase	Where to Run	Why
1-3	Foundations	Mac MPS + Kaggle	CPU/MPS sufficient. Kaggle for any GPU exercise.
4-6	Deep Learning	Kaggle (primary) + Colab Pro	Kaggle 30hr/week handles most CNN/Transformer experiments. Upgrade to Colab Pro if blocked.
7-9	Systems & NLP	Colab Pro + Lightning AI	Longer training runs. Persistent workspace matters.
10-11	Fine-tuning LLMs	Vast.ai + RunPod	Need A100 80GB for LoRA/QLoRA fine-tuning of 7B+ models. Budget ~$20-50/month.
12-13	Capstone Projects	RunPod + Lambda	Reliability matters for multi-day runs.

Monthly compute budget estimate:

Months 1-6: ₹0-899/month (Kaggle free + optional Colab Pro)
Months 7-9: ₹899-2,000/month (Colab Pro + occasional Vast.ai)
Months 10-13: ₹2,000-6,000/month (Vast.ai/RunPod for serious fine-tuning)

What Most People Get Wrong¶

Spending time on compute when they should be spending time on code. The single biggest mistake in the first 3 months: staring at GPU availability dashboards instead of writing training loops. A slow CPU training run that completes in 20 minutes teaches you more than a fast GPU run you spent 2 hours setting up.

Not checkpointing. Every training run over 30 minutes needs checkpointing. On Colab, you lose everything if the session dies without saving to Drive. Always:

# Save checkpoint every N epochs
if epoch % save_every == 0:
    torch.save({
        'epoch': epoch,
        'model_state_dict': model.state_dict(),
        'optimizer_state_dict': optimizer.state_dict(),
        'loss': loss,
    }, f'checkpoint_epoch_{epoch}.pt')

Paying AWS/GCP rates. A p3.2xlarge on AWS (V100 16GB) costs $3.06/hour on-demand. A Vast.ai V100 costs $0.15-0.30/hour. Unless your company is paying or you have a specific reason (compliance, specific region), avoid AWS/GCP for learning workloads.

Return to README.md · Previous: 01_python_environment_setup.md · Next: 03_development_environment.md