03 — Python Performance Literacy

“You will write ~80% of your kernel harnesses in Python. The GIL, asyncio, and NumPy strides are not optional literacy — they are the language you’ll be debugging in at 2 AM.”

Why This Chapter Is Not “Learn Python”

You already write production Python at Zoho. What you probably don’t have (unless you’ve built specifically-async services): a mechanical model of the interpreter, the GIL, asyncio’s event loop, and NumPy’s memory model. These are the exact skills vLLM’s Python layer, torch.profiler, and your own benchmark harnesses will demand.

The secret is that inference engines are Python at the top and C++/CUDA at the bottom, and the interesting bugs live at the boundary. Understanding both sides is what lets you diagnose why your throughput drops 30% when the async scheduler misbehaves.

The GIL, Properly

The Global Interpreter Lock is not “one thread at a time.” It is: one thread at a time can execute Python bytecode. Native code (NumPy, PyTorch, CUDA kernel launches) releases the GIL. This is why:

  • torch.matmul on the GPU can run concurrently with Python code in another thread — the GIL is released while the CUDA kernel runs.

  • time.sleep() releases the GIL (it’s a syscall). CPU-bound Python loops do not.

  • Multi-threaded Python is fine for I/O-bound work (the GIL is released during blocking syscalls).

  • Multi-threaded Python is useless for CPU-bound pure-Python work. Use multiprocessing or, better, offload to native code.

PEP 703 and the free-threaded build (Python 3.13+): experimental no-GIL Python is landing. In 3.13 it’s --disable-gil at build time. In 3.14 (October 2025) it’s a supported option. This changes the calculus — but not for the next 12 months of your journey; assume GIL exists.

Reading:

asyncio at Engine-Loop Depth

You will write async code every phase from here on. vLLM’s V1 architecture is an async engine. Your Zoho harnesses are async. Your mini-engine capstone will be async. This is direct-transfer territory.

Core mental model:

 Event Loop
     │
     ├── Task 1: awaits I/O    ───► suspended, callback registered
     ├── Task 2: awaits I/O    ───► suspended
     └── Task 3: CPU-bound     ───► blocks the loop! Bad.

The event loop is a single-threaded scheduler that runs coroutines cooperatively. A coroutine that never awaits blocks every other task. This is the #1 async bug: a synchronous call (a heavy JSON parse, a pandas.read_csv, a synchronous SQL query) inside an async function.

Idioms you must own

Concurrency, not parallelism:

# Sequential (bad):
r1 = await fetch(url1)
r2 = await fetch(url2)

# Concurrent (good):
r1, r2 = await asyncio.gather(fetch(url1), fetch(url2))

Timeouts (never forget these in production):

async with asyncio.timeout(5.0):  # 3.11+
    result = await slow_call()

Streaming with async generators (this is exactly the token-streaming pattern):

async def stream_tokens():
    async for chunk in engine.generate(prompt):
        yield chunk

Offloading CPU work:

result = await asyncio.to_thread(cpu_heavy_function, arg)
# Or, for actual parallelism, use a ProcessPoolExecutor.

Backpressure via bounded queues:

queue = asyncio.Queue(maxsize=100)  # Producers block when full.

Reading and study

NumPy Memory Model

NumPy arrays are (data pointer, dtype, shape, strides). Everything downstream — PyTorch tensors, JAX arrays — uses the same abstraction. Master these five concepts:

1. Strides. A stride is “how many bytes to skip to get to the next element in this dimension.” For a C-contiguous float32 matrix of shape (M, N), strides are (4*N, 4).

import numpy as np
a = np.arange(12).reshape(3, 4)
print(a.strides)  # (16, 4) for int32: 16 bytes per row, 4 bytes per column.

2. Views vs copies. Slicing (a[1:, :]) returns a view — same underlying buffer, different strides. Fancy indexing (a[[0, 2]]) returns a copy. Transpose is a view. Reshape might be a view or a copy depending on contiguity.

a = np.zeros((4, 4))
b = a[::2]        # View. Modifying b modifies a.
c = a.T           # View, but not C-contiguous!
d = a.reshape(16) # View iff contiguous.
e = a.copy()      # Guaranteed new buffer.

3. Contiguity. a.flags['C_CONTIGUOUS'] and a.flags['F_CONTIGUOUS']. Many ops silently fall back to slow paths (or make a copy) if not contiguous. np.ascontiguousarray(a) forces contiguity.

4. Broadcasting. The rule is right-align shapes, dimensions of size 1 stretch. Broadcasting is a stride manipulation — no memory allocated for the stretched dimension. This means it’s essentially free.

5. np.einsum. The universal contraction. Learn it — it will teach you to think in indices, which is exactly the mindset for CUDA kernel dimensions.

# Batched matmul via einsum:
C = np.einsum('bik,bkj->bij', A, B)
# Attention scores:
scores = np.einsum('bqhd,bkhd->bhqk', Q, K)

PyTorch parallel

All of the above applies to torch.Tensor with tiny renames: tensor.stride(), tensor.is_contiguous(), tensor.contiguous(), torch.einsum. Plus:

  • tensor.view() vs tensor.reshape(): view requires contiguity and never copies; reshape copies if needed. Use view in performance-sensitive paths so a silent copy shows up as an error, not a slowdown.

  • tensor.to(device, non_blocking=True) for async host→device transfer overlapped with compute.

  • Pinned memory (pin_memory=True in DataLoader): faster host→device copies, but locks pages, so don’t overuse.

Profiling Python and PyTorch

cProfile and snakeviz

For pure-Python bottlenecks:

python -m cProfile -o out.prof script.py
snakeviz out.prof

Only useful when the hot code is pure Python. Useless for finding “the CUDA kernel is slow” — you’ll see the launch as a fast call and miss the story.

py-spy (the one you’ll actually use)

Sampling profiler that attaches to a running process. Zero code changes. This is a superpower for production debugging.

pip install py-spy
py-spy top --pid 12345                         # Live top-like view.
py-spy record -o profile.svg --pid 12345       # Flamegraph.
py-spy dump --pid 12345                        # Stack traces of all threads.

Repo: https://github.com/benfred/py-spy — 20k+ stars, actively maintained.

torch.profiler (the essential ML profiler)

Instrument the code, get a Chrome trace showing CPU ops, CUDA kernels, and their overlap.

import torch
from torch.profiler import profile, ProfilerActivity, schedule, tensorboard_trace_handler

with profile(
    activities=[ProfilerActivity.CPU, ProfilerActivity.CUDA],
    schedule=schedule(wait=1, warmup=1, active=3, repeat=1),
    on_trace_ready=tensorboard_trace_handler('./log'),
    record_shapes=True,
    profile_memory=True,
    with_stack=True,
) as prof:
    for step in range(10):
        model(input)
        prof.step()

Open the trace in chrome://tracing or Perfetto (https://ui.perfetto.dev/). This is the tool. Master it in Phase 0 so it’s second nature by Phase 3.

Reading:

nvidia-smi dmon and nvtop

At-a-glance GPU utilization. Coarse but instantly useful:

nvidia-smi dmon -s pucm      # Power, util, clocks, memory
nvtop                        # htop for GPUs

The Micro-Benchmark Discipline

When you compare two Python implementations of the same thing, do this:

import torch, time

def bench(fn, warmup=10, iters=100):
    # Warmup — JIT, allocator, cache priming.
    for _ in range(warmup):
        fn()
    torch.cuda.synchronize()
    # Time.
    start = torch.cuda.Event(enable_timing=True)
    end = torch.cuda.Event(enable_timing=True)
    start.record()
    for _ in range(iters):
        fn()
    end.record()
    torch.cuda.synchronize()
    return start.elapsed_time(end) / iters   # ms per iter

Rules you break at your peril:

  1. Always warmup. First-call effects (JIT, cuBLAS handle init, cache misses) are massive.

  2. Always torch.cuda.synchronize() before and after timing GPU work.

  3. Use cuda.Event, not time.perf_counter for GPU timing — events are on-device markers, wall-clock timers include Python overhead.

  4. Lock GPU clocks on your dev box (sudo nvidia-smi -lgc <freq>) to defeat thermal throttling variance. Unlock (-rgc) when done.

  5. Report p50, p95, p99 — not means. LLM serving has heavy tails; means lie.

Adopt this discipline now on Python micro-benchmarks; it’s the same discipline for kernel benchmarks in Phase 2.

Exit Deliverable

  1. async_scratch.py: a small script that spawns 100 mock “requests” against an async worker with a bounded queue and streams results back. This is a Phase 4 mini-engine warmup.

  2. numpy_strides.md: your write-up of a hands-on experiment showing (a) transpose is a view, (b) reshape after transpose forces a copy, (c) broadcasting doesn’t allocate, with .strides and memory-address evidence.

  3. bench_harness.py: your reusable benchmark harness (warmup, sync, event-timed, p50/p95/p99 report). You’ll paste this into every project going forward.

  4. A torch.profiler trace of any small model forward pass, opened in Perfetto, with a paragraph on what you noticed.