RAG Cost Architecture: Ingestion, Chunking & Vector Retrieval Economics
By TokenMath Editorial & Engineering Team · Reviewed · Methodology
Retrieval-Augmented Generation (RAG) splits operational expenses into two distinct regimes: one-time corpus embedding and recurring context amplification. Without meticulous budgeting, vector database IOPS and multi-chunk prompt bloat often exceed baseline LLM token costs by an order of magnitude.
1. The Fundamental Two-Phase Cost Equation
RAG architectures separate computational expenditures into two fundamentally decoupled billing phases:
Occurs during initial onboarding or scheduled batch refreshes. The total expense depends on raw document size, chunk overlap percentage, embedding API pricing, and vector database write fees.
Recurring cost accrued on every user search or agent tool execution. Includes user prompt embedding, vector index similarity search, cross-encoder neural reranking, and generation LLM input tokens.
2. Vector Dimensionality & Memory Footprint Mathematics
A common budgeting error is overlooking raw vector storage and index memory. In vector databases (Pinecone, Qdrant, Weaviate, pgvector), every dense vector is represented as a high-dimensional array of 32-bit floating-point numbers (float32).
Note: Hierarchical Navigable Small World (HNSW) graph indices require an additional 20% to 50% memory overhead to store graph edge lists (M parameter) and document metadata payloads.
| Embedding Model | Provider | Dimensions | Bytes / Vector | Price / 1M Tokens | Recommended Use Case |
|---|---|---|---|---|---|
| text-embedding-3-small | OpenAI | 1536 | 6.14 KB | $0.02 | High-throughput general search, budget pipelines |
| text-embedding-3-large | OpenAI | 3072 | 12.29 KB | $0.13 | Complex semantic distinction, legal and medical corpora |
| voyage-3 | Voyage AI | 1024 | 4.09 KB | $0.12 | Long-context documents, technical code retrieval |
| embed-english-v3.0 | Cohere | 1024 | 4.09 KB | $0.10 | Short chunk enterprise search, compression |
3. Cloud Vector Database Infrastructure Cost Comparison
Storing and querying vectors across managed infrastructure incurs distinct cost structures depending on whether you choose serverless pay-per-read or provisioned memory clusters:
| Platform | Write / Ingest Rate | Query / Read Rate | Storage / GB / Mo | Architectural Trade-Off |
|---|---|---|---|---|
| Pinecone Serverless | $2.00 / 1M Write Units | $0.25 / 1M Read Units (RU) | $0.33 / GB / month | Zero cold start, managed HNSW partitioning |
| Weaviate Cloud | Included in cluster compute | ~$0.085 / 1M vector queries | ~$0.20 / GB / month | Hybrid BM25 + Vector native engine |
| Qdrant Cloud | Included in cluster tier | Flat cluster fee (~$25/mo for 1M vectors) | ~$0.30 / GB / month | High payload filtering performance, Rust runtime |
| PostgreSQL + pgvector (RDS/Neon) | DB compute / IOPS bound | Shared DB IOPS (Compute dependent) | $0.10 - $0.25 / GB / month | HNSW index must fit in shared_buffers RAM |
4. The Context Amplification Trap (Top-K Multiplication)
When estimating per-query cost, developers frequently budget for the user's 50-token query and an anticipated 300-token answer. However, the retrieval phase injects K document chunks directly into the prompt.
Suppose your system prompt is 200 tokens, the user question is 80 tokens, and you retrieve Top-K = 5 chunks at an average chunk size of 500 tokens:
At 500,000 monthly queries, that retrieved context represents 1.25 billion input tokens per month ($2,500/mo on Claude Sonnet 5 or $5,000/mo on GPT-5.6 Sol)!
5. Two-Stage Retrieval & Neural Reranking Economics
Instead of dumping 15–20 weakly relevant chunks directly into the generation LLM, high-efficiency architectures utilize Two-Stage Retrieval:
- Stage 1 (High Recall): Retrieve Top-25 candidates cheaply using hybrid BM25 + dense vector similarity ($0.000005 per query).
- Stage 2 (High Precision Rerank): Pass the 25 candidates through a cross-encoder model (e.g.
Cohere Rerank 3at $1.00 / 1,000 queries orVoyage Rerank). - Generation Step: Pass only the Top-3 reranked chunks (1,500 tokens) to the frontier LLM instead of 25 chunks (12,500 tokens).
Sending 12,500 tokens to Claude Sonnet 5 ($2.00/1M) costs $0.025 per query. Trimming context to 1,500 tokens via Cohere Rerank costs $0.003 (LLM) + $0.001 (Reranker) = $0.004 per query. That is an 84% net cost reduction while simultaneously boosting answer accuracy by eliminating hallucination-inducing distractor chunks.
6. Worked Production Architecture: 100k Documents, 250k Queries / Month
Below is a real-world financial cost projection for an enterprise support RAG system indexing 100,000 PDF pages (50,000,000 tokens) serving 250,000 user inquiries monthly:
| Pipeline Component | Workload Specifications | Unit Rate | Monthly Cost |
|---|---|---|---|
| Corpus Ingestion | 50M tokens (text-embedding-3-small) amortized quarterly | $0.02 / 1M | $0.33 / mo |
| Vector Database Storage | 100k chunks × 1536-dim + index (~1.2 GB) on Pinecone | $0.33 / GB / mo | $0.40 / mo |
| Query Embedding | 250k queries × 60 tokens = 15M query tokens | $0.02 / 1M | $0.30 / mo |
| Vector DB Similarity Searches | 250k Read Units (Pinecone Serverless) | $0.25 / 1M RU | $0.06 / mo |
| Neural Cross-Encoder Reranking | 250k searches reranking top-20 candidates to top-3 | $1.00 / 1k req | $250.00 / mo |
| LLM Generation (Gemini 2.5 Flash) | 250k × (1,700 in @ $0.30 / 300 out @ $2.50) | $0.30 in / $2.50 out | $315.00 / mo |
| Total Monthly Budget | Fully configured two-stage enterprise pipeline | Unit: $0.00227 | $566.09 / mo |
7. Production Optimization Checklist
- Match Embedding Dimensionality to Corpus Complexity: Avoid 3,072-dimension embeddings for generic English FAQs when 1,024 or 1,536 dimensions provide comparable recall at 50% to 66% lower RAM footprint.
- Implement Prompt Caching on Shared Preamble: If all user queries share a 2,000-token system instruction and static company terminology glossary, cache it to receive 75% to 90% read discounts on every query.
- Compress Retrieved Payloads: Clean HTML/Markdown tags and unnecessary metadata before passing retrieved chunks to the LLM generation prompt.
- Calibrate Chunk Overlap: Limit overlap to 10%–15% of chunk size. Overlaps exceeding 25% cause redundant duplicated text to occupy costly LLM context slots.
Simulate Your RAG Pipeline TCO
Input your corpus size, query volume, chunking parameters, and vector DB choice in our interactive RAG calculator.