Input vs Output Tokens: Why Output Costs 3x to 5x More
Examine every AI pricing sheet and you will discover a universal truth: generating output tokens costs between 300% and 500% more than processing input tokens. Here is the mathematical and hardware reality behind this asymmetry.
1. The Two Phases of Transformer Inference: Prefill vs Decode
To understand asymmetric token pricing, you must understand how modern transformer Large Language Models process queries on GPU silicon. Inference consists of two computationally distinct phases:
All prompt tokens are processed simultaneously in parallel. GPUs operate at peak tensor core FLOPS efficiency because matrix multiplications can saturate memory bandwidth across all streaming multiprocessors.
Output tokens are generated sequentially, one token per forward pass. The GPU must load hundreds of gigabytes of model weights from High Bandwidth Memory (HBM) for every single generated token.
2. The Memory Bandwidth Bottleneck (Roofline Model)
During token generation, modern GPUs like the NVIDIA H100 SXM5 (3.35 TB/s memory bandwidth) or B200 (8 TB/s) are not compute-bound; they are memory bandwidth-bound.
When generating a single token for a 70-billion parameter model quantized to FP8 (70 GB of weights), the GPU cluster must read 70 GB of data through the memory bus just to compute one token. If you generate 100 output tokens, that is 7,000 GB (7 TB) of memory transfers!
In contrast, during prompt prefill, a 4,000-token prompt reads those same 70 GB of weights once and computes all 4,000 tokens in parallel. The compute-to-memory-access operational intensity is 4,000 times higher.
3. Key-Value (KV) Cache Memory Pressure
Every token in a conversation generates attention keys and values ($K$ and $V$) that must be preserved in GPU VRAM for all subsequent tokens. As an output sequence expands:
- Each active request reserves a dedicated slice of GPU HBM for its growing KV cache.
- When VRAM fills up, the server can no longer batch other concurrent requests, drastically lowering the provider's overall GPU utilization.
- Techniques like PagedAttention (vLLM) and Multi-Head Latent Attention (DeepSeek MLA) reduce this overhead, but output generation remains the limiting factor for concurrent inference concurrency.
4. FinOps Implications: Architecting for High Input, Low Output
Because input is cheap and output is expensive, the most cost-efficient AI architectures maximize input context while minimizing output generation:
- Structured Extraction: Instruct models to return boolean flags or compact enums rather than verbose natural language explanations.
- RAG Compression: Pass rich reference documents in the prompt and ask for a 2-sentence synthesis.
- Prompt Caching Synergy: Combining high input with 90% prompt caching discounts drops input costs to pennies while protecting your budget from expensive decode cycles.
Simulate Input vs Output Cost Ratios
Test custom prompt and output sizes across all 30 models.