09 — Cost Modeling: $/1M Tokens, Tiering, and Fine-Tune-Small vs Prompt-Big

The technical answer decides how to serve. The cost model decides whether to serve. Most inference engineers know the first and are guessed-at on the second, and that gap is why the same job title pays 30-50% differently. This doc closes that gap. The core skill is the ability to walk into any deployment discussion and quote, defensibly, the $/1M-tokens number under three different assumptions in under five minutes.

1. The Fundamental Formula

$/1M tokens = (GPU-hour cost) / (measured goodput in tokens/hour) × 1,000,000

Goodput is not throughput. Goodput = throughput that meets SLO. Tokens generated that violated the latency contract are not sold; they might even be refunded. Report cost against goodput or you are lying to yourself.

Three modifiers everyone forgets:

  1. Utilization. A GPU idle at night still bills. Divide by realistic duty cycle (usually 30–60% for interactive; higher for batch).

  2. Replica floor. If you keep minReplicas=2 for HA, small deployments effectively double.

  3. Amortized cold-start. If your fleet churns replicas often, you are paying for warmup tokens that never sold.

Honest formula:

effective $/1M tokens = (GPU-hourly-rate × replicas × 24) / (peak_goodput × duty_cycle_fraction) × 1e6 / hours

2. Worked Numbers (2026 rates)

Using verified Q2 2026 rates (see doc 10 for the full table).

2.1 On a rented H100 SXM

  • Rate: $2.50/hr (neocloud mid-market)

  • Model: Llama 3.3 70B fp8, TP=4 → cost per replica-hour = $10.00

  • Measured goodput at target SLO: 3000 tok/s aggregate under batched agent load

  • Tokens per replica-hour: 3000 × 3600 = 10.8M

  • Cost: $10.00 / 10.8M × 1e6 = $0.93 per 1M tokens at 100% duty

  • At 60% realistic duty: $1.55 per 1M tokens

Compare to OpenAI GPT-4.1 at ~$5/1M input, $15/1M output as of Q2 2026 (public rate card). Your on-prem 70B is 5–10x cheaper per token at the cost of some quality bar difference.

2.2 On owned on-prem H100

  • Cap-ex: $28,000 per H100 × 8 = $224,000

  • 3-year amortization: $6,222/month = $8.53/hour cluster = $1.07/hour per GPU

  • Add power: 8 × 700W × 1.4 PUE × $0.12/kWh × 24 = $22.58/day = $0.94/hour cluster

  • Rack/network/cooling/staffing amortized: ~$500/month = $0.69/hour

  • Total: ~$10.16/hour cluster = $1.27 per GPU-hour

  • Same workload as above: $5.08/replica-hour on 4 GPUs, 10.8M tokens/hr

  • $0.47 per 1M tokens at 100% duty, $0.78 per 1M tokens at 60%

Owned hardware is ~2x cheaper per token than rented at 3-year horizon. For on-prem enterprise deals with predictable load, the math favors capex.

2.3 On MI300X (Vultr)

  • Rate: $1.85/hr

  • 70B fp8 on ONE MI300X (fits comfortably): cost per replica-hour = $1.85

  • Measured goodput: ~1500 tok/s per replica at target SLO (single-GPU can’t push same batch as TP=4 on H100)

  • Tokens per replica-hour: 5.4M

  • Cost: $1.85 / 5.4M × 1e6 = $0.34 per 1M tokens at 100% duty

Subject to the ROCm maturity caveats (doc 08), MI300X at Vultr’s price point is the cheapest 70B serving in the market right now. Do the benchmark to confirm.

2.4 On L40S (four independent replicas of Qwen 14B fp8)

  • Rate: $1.00/hr per L40S, cluster = $4.00/hr

  • Aggregate goodput: 3000 tok/s across four replicas at 200 concurrent users

  • Tokens per hour: 10.8M

  • Cost: $0.37 per 1M tokens at 100% duty

This is nearly identical cost per token to the H100 70B, but for a 14B model. The interpretation: if your workload’s quality bar is met by 14B (and it often is for tool-heavy agents), L40S is your economically dominant strategy. This is the fine-tune-small argument made numerical.

3. Batch Tier vs Interactive Tier

The same model on the same hardware costs different $/token in the two tiers because of goodput.

Dimension

Interactive

Batch

SLO

TTFT p95 = 1s, ITL p95 = 100ms

E2E within 1 hour

max-num-seqs

32-64 (keep ITL)

256-512 (max throughput)

max-num-batched-tokens

4k-8k

32k+

Chunked prefill

Aggressive

Off (throughput > TTFT)

Utilization

30-60% (bursty)

90-99% (queue-drained)

$/1M tokens

2-3x batch

Baseline

The operational move: tier your workloads. Route heavy async jobs (bulk summarization, embedding backfill, offline evaluation, agentic batch runs) to a batch-tier deployment with different flags. The same H100 fleet can serve interactive at $1.55/1M and batch at $0.55/1M by tuning knobs, and the customer sees two SKUs with different price points — which is exactly how OpenAI and Anthropic sell it. Copy the shape; the shape is right.

3.1 Batch API pattern

vLLM has a batch endpoint; if not using it, the pattern is: queue requests in Redis or Kafka, drain into a batch-tier vLLM replica at maximum max-num-seqs, return results asynchronously (webhook, polling endpoint, or object-store write). Latency-insensitive, throughput-maximum. Standard for evaluations, backfills, and any “we’ll email you when it’s ready” workflow.

4. When Fine-Tune-Small Beats Prompt-Big

The biggest lever in the cost model is not knob-tuning; it is choosing a smaller model.

4.1 The break-even calculation

prompt-big cost per task     = tokens_prompt_big × price_per_token_frontier
fine-tune-small cost per task = tokens_small × price_per_token_small + amortized_finetune_cost

amortized_finetune_cost = one_off_finetune_cost / expected_task_count

Example: extract structured data from a support ticket.

  • Prompt-big: GPT-4.1 with a 3k-token prompt + 500-token output = 3500 tokens × $15/1M avg = $0.053 per task

  • Fine-tune-small: Qwen 2.5 7B fp8 fine-tuned on 5000 labeled examples. Fine-tune cost ~$200 (one-time), inference ~$0.001 per task at 500 total tokens × $0.40/1M

  • Break-even: 200 / (0.053 - 0.001) = 3,850 tasks

Any production workload doing >4000 identical-shape tasks per year is cheaper to fine-tune. This is not a close call for Zoho CRM workloads, which are almost by definition high-volume, narrow-shape. The fine-tune-small argument is almost always right at Zoho scale.

4.2 Where prompt-big wins

  • Low volume, high variance tasks. R&D exploration, one-off summarization.

  • Long-tail tasks where you cannot collect training data.

  • Quality-mandatory tasks where 7B fp8 quality is unacceptable and you cannot fine-tune above 7B economically.

  • Reasoning-mandatory tasks where thinking-model quality (o-family, GPT-5, Claude Opus 4.6) is genuinely required. Do the eval before assuming.

4.3 The stealth cost of prompt-big

A hidden cost people miss: prompt-big means every token of a growing agent conversation costs frontier-model money. A 20-turn agent conversation on GPT-4.1 with 500 tokens of history per turn easily hits $0.50-$1.00 per session. At 500 daily active users, that’s $250-$500/day = $90k-$180k/year in per-user API bills. Compare to $16-$25k/year for owned or amortized on-prem 14B fp8 hardware serving the same volume. The math is not close.

This is the exact Zoho argument for on-prem inference. Bring these numbers to the strategy meeting.

5. Cost Model Components You Cannot Skip

The full picture, in order of magnitude for typical enterprise deployments:

  1. GPU compute (60-80% of TCO). Rented $/hr or owned amortized.

  2. Power (10-20%). GPU × TDP × PUE × electricity rate × 24 hours.

  3. Networking egress (0-15%). Cloud can bite you here; on-prem is nearly free.

  4. Storage (5-10%). Model weights, KV cache spill, logs, traces. Object store + fast NVMe tier.

  5. Observability (2-5%). Prometheus, Grafana, Tempo. Free if self-hosted.

  6. Human operations (varies). This is not zero. A production LLM fleet needs 0.5-2 FTEs of on-call.

  7. Cold-start waste (2-10%). Warmup tokens don’t sell but pay for themselves in avoided SLO breaches.

  8. Software licenses (varies). Enterprise support contracts for vLLM (via Red Hat / IBM), NVIDIA AI Enterprise, monitoring vendors.

A line-item cost model with these eight rows, updated quarterly, is what CFO-level LLM conversations require. Build the spreadsheet template once, reuse it forever.

6. Speculative Decoding: The Cost Multiplier That Can Bite

Speculative decoding trades draft-model compute for target-model bandwidth. It wins big at low batch (latency-sensitive) and can lose at high batch (draft compute steals from useful work). Numbers:

  • Break-even acceptance rate: usually 60-70%. Below that, spec-decode makes you slower.

  • EAGLE-2/3 on Llama-70B in production: 3-5x speedup for interactive TTFT, well above break-even.

  • Naive draft-model speculation on aligned model pair: 1.5-2.5x, still worth it.

  • MTP (multi-token prediction as in DeepSeek V3): 1.7-2x, and free at inference because it was trained in.

Always report vllm:spec_decode_acceptance_rate in your dashboard. If it drifts below 60%, disable spec-decode; you are paying compute for nothing. This is a real production knob — do not set-and-forget.

7. The KV Cache As A Money-Maker (Prefix Caching Economics)

With 75% prefix-cache hit rate on agentic traffic (SGLang RadixAttention verified numbers, doc 05):

  • Prefill FLOPs saved: 75% × (prefill_share_of_total_compute_time) — for agentic 3-5x share, this is 50-60% of total compute

  • Cost implication: prefix caching alone can halve your $/1M-tokens on agent workloads

  • Zero risk. Zero quality delta. Purely a flag in vLLM/SGLang.

This is the highest-ROI single optimization in the entire production doc. If you have not measured prefix cache hit rate on Zoho agent traffic, do it this week. It is the most credible “technical proof of ROI” a mid-career engineer can produce inside a large company.

8. Reserved vs Spot vs On-Demand

GPU rental costs vary 3-5x by commitment level:

Commit

Rate

When

On-demand

1x (baseline)

Bursty dev; unpredictable workloads

Reserved 1yr

0.5-0.7x

Steady baseline capacity

Reserved 3yr

0.3-0.5x

Long-committed production

Spot / preemptible

0.2-0.4x

Batch tier, checkpoint-tolerant

Owned + 3yr amortization

0.3-0.5x

On-prem baseline

Production pattern: reserve the baseline, burst on-demand, batch on spot. Same story as classic cloud CPU capacity, applied to GPUs. Track your baseline utilization for 30 days, reserve at ~80% of baseline, burst above.

9. Cost Modeling Anti-Patterns

  1. Reporting cost on peak, not p95 sustained. Peak numbers are marketing. Sustained numbers are decisions.

  2. Ignoring the minReplicas floor for HA — a 2-replica floor doubles small deployment costs.

  3. Assuming 100% duty cycle. Reality for interactive is 30-60%. Batch closer to 90%.

  4. Forgetting KV pool sizing. Under-provisioned KV = preemption storms = throughput collapse = cost triples silently.

  5. Treating spec-decode as free. It has a compute cost that must be justified by acceptance rate.

  6. Cost comparisons across dtypes without quality checks. “fp4 is 2x cheaper” is meaningless if perplexity ballooned. Every cost delta gets a quality delta measurement.

10. Zoho angle

Every month, produce a one-slide internal cost report:

  • $/1M tokens by tenant tier

  • Prefix-cache hit rate by workload

  • Batch vs interactive tier utilization

  • Marginal cost of the next-100-users, per workload

  • Comparison: owned vs the equivalent frontier-API spend for the same task volume

That slide, in front of enough VPs, becomes your leverage. It is the closest thing to a “P&L for inference” that exists in most orgs, and shipping it monthly is a distinctive career move.

11. Reading list

  • Baseten’s public cost-per-token benchmarks — imitable methodology

  • Together AI pricing page — what the market accepts as fair per-model

  • OpenAI, Anthropic public rate cards — the ceiling you are competing against

  • Character.AI blog on cost reduction (sub-cent-per-hour) — the aspirational floor

  • “How to Scale Your Model” (Google DeepMind) — chapter on roofline economics

  • Chip Huyen, AI Engineering — chapter on model selection cost tradeoffs

12. Exit test

  1. Given a workload description and a hardware SKU, produce a $/1M-tokens number with a defensible duty-cycle assumption, in five minutes.

  2. For a hypothetical Zoho CRM agent workload of 500 daily active users at 20 turns each, produce a side-by-side spreadsheet of: (a) GPT-4.1 API cost, (b) rented on-prem H100 cost, (c) owned on-prem H100 cost, (d) fine-tuned Qwen 14B on L40S cost — with all assumptions listed.

  3. Explain when speculative decoding stops being economical and back it with acceptance-rate math.

  4. Present a one-slide monthly cost report for a hypothetical Zoho inference platform: $/1M-tokens by tier, prefix-cache hit rate, comparative-to-frontier-API dollars saved.