04 — Tools and Libraries Canon

The Python ML ecosystem is dense, partially overlapping, and changing fast enough that “standard tooling” from 2020 is becoming legacy tooling in 2026. This file maps the essential libraries, their current stable versions, what they’re actually for, their failure modes, and — where relevant — whether they’re being replaced by something better. Install only what you need for each phase; don’t set up the full stack on day one.

What most people get wrong: They install everything into one environment, create dependency conflicts, spend a day debugging, and then stop touching the environment. Use separate environments per project. The setup cost is 5 minutes. The debugging cost of a polluted environment is hours.

---

The Core Stack (Install Order Matters)

---

1. Python

Current recommended version: 3.11.x (as of mid-2026)

Python 3.12 is stable but some ML libraries (particularly older CUDA-adjacent packages) still lag on 3.12 support. Python 3.10 is the safe minimum for PyTorch 2.x. Python 3.11 is the sweet spot: full async improvements, faster CPython (~25% speedup over 3.10), and full ecosystem support.

Install via: uv or pyenv (do NOT use the system Python for ML work)

# Using uv (recommended)
uv python install 3.11

# Using pyenv (alternative)
pyenv install 3.11.9
pyenv global 3.11.9

⚠️ Pitfall: macOS ships with Python 2.7 or 3.x system Python. Never pip install into it. You will corrupt system tools and spend a weekend restoring your PATH.

---

2. NumPy

Current version: 2.x (NumPy 2.0 released June 2024 — breaking changes from 1.x) Purpose: N-dimensional array operations, the foundation of the entire scientific Python stack. Every ML library operates on NumPy arrays or tensors that convert to them.

Critical pitfall: NumPy 2.0 introduced breaking changes. If you’re using older libraries (pre-2024), pin to numpy<2.0 to avoid incompatibilities.

uv pip install numpy
# Or pin: uv pip install "numpy<2.0"

Is it being replaced? Not really. JAX’s array API is compatible, and PyTorch tensors inter-operate. NumPy remains the lingua franca.

---

3. Pandas

Current version: 2.2.x Purpose: Tabular data manipulation. Loading CSVs, feature engineering, groupby operations, joins, time series resampling.

Critical pitfall: Pandas can silently do column operations in-place or return copies depending on context (SettingWithCopyWarning). The Pandas 2.0 Copy-on-Write (CoW) mode is now the default and resolves this — but CoW semantics differ from 1.x behavior.

uv pip install pandas

Is it being replaced? Partially. Polars (Rust-based) is 5-30x faster for large DataFrames and has a cleaner API. Community recommendation: use Pandas for datasets under ~1GB, Polars for larger data pipelines. Polars is not a drop-in replacement — it has a different API. The transition is happening but Pandas isn’t dead.

uv pip install polars  # If working with large tabular data

---

4. Matplotlib + Seaborn

Current version: Matplotlib 3.9.x, Seaborn 0.13.x Purpose: Data visualization. Matplotlib is the low-level engine; Seaborn is a statistical visualization layer on top.

uv pip install matplotlib seaborn

Alternatives: Plotly for interactive plots, Altair for declarative grammar-of-graphics style. For quick training curve visualization, use W&B or TensorBoard instead of rolling your own Matplotlib plots.

---

5. Scikit-learn

Current version: 1.5.x Purpose: Classical ML algorithms (linear models, SVMs, decision trees, random forests, gradient boosting, clustering, dimensionality reduction), preprocessing (scalers, encoders), model selection (cross-validation, grid search), pipelines.

uv pip install scikit-learn

Critical pitfall: Scikit-learn uses NumPy arrays natively. Its Pipeline API is excellent but requires understanding fit/transform/predict semantics. The most common bug: fitting the scaler on the full dataset before train/test split (data leakage).

Is it being replaced? No. Still the standard for tabular ML and preprocessing. For gradient boosting specifically, XGBoost/LightGBM/CatBoost are faster and more accurate.

---

6. PyTorch

Current version: 2.4.x (as of mid-2026) Purpose: The dominant deep learning framework. Dynamic computation graphs, autograd, GPU acceleration via CUDA (or MPS on Apple Silicon). The framework of choice for research and increasingly production.

# CPU only
uv pip install torch torchvision torchaudio

# CUDA 12.1 (Linux/Windows)
pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu121

# Apple Silicon MPS (macOS)
pip install torch torchvision torchaudio  # MPS support is in the base package

CUDA compatibility matrix (critical):

PyTorch	CUDA	Python
2.4.x	12.1, 12.4	3.9-3.12
2.3.x	11.8, 12.1	3.8-3.12
2.0.x	11.7, 11.8	3.8-3.11

⚠️ Pitfall: Don’t install PyTorch with pip install torch without specifying the CUDA index URL if you need CUDA. The PyPI version only includes CPU. Always verify with:

import torch; print(torch.cuda.is_available())

Is it replacing TensorFlow? In research: yes, effectively. PyTorch has 85%+ adoption in ML papers. TensorFlow is still used in production at Google-scale, but for a learner, PyTorch is the correct choice.

---

7. HuggingFace Ecosystem

Purpose: The unified API for pretrained models, datasets, and training utilities for NLP, vision, audio, and multimodal tasks.

Core packages:

Package	Version	Purpose
transformers	4.42.x	Load/fine-tune/run pretrained models (BERT, GPT, LLaMA, etc.)
datasets	2.20.x	Load/process datasets (HF Hub, local, streaming)
tokenizers	0.19.x	Fast tokenization (Rust-backed)
peft	0.11.x	LoRA, QLoRA, prompt tuning (parameter-efficient fine-tuning)
trl	0.9.x	SFT, RLHF, DPO training loops for LLMs
accelerate	0.31.x	Multi-GPU/TPU training, device-agnostic code
evaluate	0.4.x	Metric computation (BLEU, ROUGE, accuracy, etc.)
diffusers	0.29.x	Stable Diffusion, DALL-E inference and fine-tuning

uv pip install transformers datasets tokenizers peft trl accelerate evaluate

Critical pitfall: transformers model weights are cached in ~/.cache/huggingface/hub/ by default. This fills up fast — a 7B model is ~13GB. Set HF_HOME to a drive with sufficient space.

---

8. Gradient Boosting Libraries

Purpose: Best-in-class tabular data performance. XGBoost and LightGBM consistently win Kaggle tabular competitions over deep learning.

Library	Version	Speed	Notes
xgboost	2.1.x	Fast	Gold standard, GPU support
lightgbm	4.4.x	Faster	Better on large datasets
catboost	1.2.x	Moderate	Best for categorical features natively

uv pip install xgboost lightgbm catboost

---

9. Experiment Tracking

Purpose: Track hyperparameters, metrics, model checkpoints, and reproduce experiments. Essential for anything beyond a toy model.

Weights & Biases (W&B):

• Version: 0.17.x

• Free tier for individuals and academics

• Best UI, automatic system metrics, artifact versioning, sweeps for hyperparameter optimization

uv pip install wandb
wandb login  # Requires free account at wandb.ai

MLflow:

• Version: 2.14.x

• Open-source, self-hostable, no account needed

• Better for enterprise/private tracking, weaker UI than W&B

uv pip install mlflow

Which to use: W&B for learning and personal projects. MLflow when working in an organization that won’t allow external logging services. Don’t use both simultaneously unless you have a reason.

---

10. FastAPI

Current version: 0.111.x Purpose: Building ML model serving APIs. Async, auto-generates OpenAPI docs, fast startup, Pydantic validation.

uv pip install fastapi uvicorn

Critical pitfall: Loading ML models inside request handlers (on every request) is catastrophic for latency. Load the model at startup using lifespan events.

---

11. Docker

Version: 26.x (as of 2026) Purpose: Containerize ML environments for reproducible deployment. The difference between “it works on my machine” and “it works in production.”

Not a Python package — install from docker.com.

Key concepts for ML:

• FROM pytorch/pytorch:2.4.0-cuda12.1-cudnn9-runtime — use official PyTorch base images

• Mount volumes for data (-v /local/data:/app/data)

• CUDA passthrough requires --gpus all flag and NVIDIA Container Toolkit

---

12. DVC (Data Version Control)

Current version: 3.51.x Purpose: Git for data and models. Track dataset versions, model artifacts, pipeline stages. Integrates with S3, GCS, Azure Blob, SSH.

uv pip install dvc
dvc init  # Inside a git repo

Is it the standard? DVC is the most-recommended open-source data versioning tool, but it has a steep learning curve. MLflow Artifacts and W&B Artifacts are simpler alternatives for experiment artifacts specifically.

---

13. Jupyter / JupyterLab

Current version: JupyterLab 4.2.x Purpose: Interactive development for exploration, visualization, and presenting analysis.

uv pip install jupyterlab
jupyter lab  # Starts server

Critical pitfall: Jupyter notebooks are NOT production code. They hide state, make testing hard, and create reproducibility issues. Use notebooks for exploration only — move code to .py modules before treating it as final.

Install nbstripout to prevent committing notebook outputs to git:

uv pip install nbstripout
nbstripout --install  # In any git repo

---

14. Ruff + Black + isort

Purpose: Code formatting and linting. Non-negotiable for maintaining readable ML code.

uv pip install ruff black isort

Note: Ruff is a Rust-based linter that replaces flake8, isort, and many other tools in one binary. It’s 10-100x faster than the tools it replaces. As of 2025, Ruff is the community standard.

---

Environment Templates

Minimal ML environment (requirements.txt):

numpy>=1.24,<2.0
pandas>=2.0
scikit-learn>=1.3
torch>=2.2
matplotlib>=3.7
seaborn>=0.13
jupyter>=1.0
wandb>=0.17

Full LLM fine-tuning environment:

torch>=2.2
transformers>=4.40
datasets>=2.18
peft>=0.10
trl>=0.8
accelerate>=0.29
bitsandbytes>=0.43  # Required for QLoRA on CUDA
wandb>=0.17

---

What’s Being Actively Replaced (Flag)

Outgoing	Incoming	Status
conda (for pure Python)	uv	uv winning for pure Python
flake8 + isort + pylint	ruff	ruff is the new standard
Pandas (large data)	Polars	Transition ongoing, Pandas not dead
TensorFlow (research)	PyTorch	Effectively complete in research
FAISS only	FAISS + Qdrant/Weaviate/Chroma	Vector DBs fragmenting

---

Return to README.md · Previous: 03_papers_canon.md · Next: 05_datasets_canon.md