Prompt Caching Cost Economics: How KV Caching Saves Up to 90%
Prompt caching is the single most impactful architectural lever for slashing production LLM bills. Here is how Key-Value (KV) state reuse operates under the hood, how write premiums work, and how to structure prompts for maximum hit rates.
1. The Computational Bottleneck of Prompt Ingestion
When you send a 50,000-token prompt to an LLM, the model must compute the Key and Value attention matrices for every token in every layer before generating the very first output token.
If you run a RAG pipeline or an agentic loop where the first 45,000 tokens (system instructions, tool definitions, documentation chunks) are identical across multiple consecutive requests, recomputing those matrices is pure waste.
2. How Prompt Caching Operates
With prompt caching enabled, the provider stores the computed KV activations in fast GPU HBM (High Bandwidth Memory) or distributed flash cache. On subsequent calls matching that exact prompt prefix, the provider skips the prefill phase and immediately begins generation:
- Anthropic Claude: Requires explicit cache breakpoints (
cache_control: { type: 'ephemeral' }). Minimum 1,024 tokens. Cache lifetime is 5 minutes, refreshed automatically on every hit. - OpenAI: Automatic caching for prompts exceeding 1,024 tokens without explicit markup. Offers a 50% discount on cached input tokens.
- DeepSeek: Disk-backed KV cache offering a 90% discount ($0.028/1M vs $0.27/1M baseline).
3. The Write-Cache Break-Even Math
On providers like Anthropic that charge a write premium (e.g. $3.75/1M to write cache vs $3.00/1M standard on Sonnet 5), writing a prompt into cache incurs a 25% surcharge on the initial request.
However, subsequent cache reads cost only $0.30/1M (a 90% discount). Therefore:
In plain terms: a single cache hit immediately pays off the write surcharge and generates massive net savings on every call thereafter.