Batch API Processing: 50% Discounts, Queues & Evals
If your LLM workload doesn't require sub-second human interactive latency, you are overpaying by exactly 100%. Learn how to architect asynchronous Batch API pipelines to cut both input and output bills in half across OpenAI, Anthropic, and Google.
1. The Economics of Batch APIs: Why Providers Give 50% Off
Frontier LLM data centers experience massive diurnal demand curves. During daytime business hours in North America and Europe, interactive web APIs operate at peak capacity. At night or during regional lulls, millions of dollars worth of H100/B200 GPU clusters risk sitting idle while consuming baseload electrical power.
To flatten these demand peaks and monetize surplus GPU cycles, providers created Batch APIs (sometimes called Asynchronous Endpoints). Under this model:
- You submit a batch of requests formatted as a newline-delimited JSON (JSONL) file.
- The provider commits to completing the batch within a 24-hour SLA (in practice, jobs typically finish within 20 to 90 minutes).
- In exchange for deferring execution to off-peak compute windows, you receive an immediate, unconditional 50% discount on both input and output token rates.
2. Batch vs. Synchronous Pricing Comparison
| Model | Sync Input / 1M | Batch Input / 1M | Sync Output / 1M | Batch Output / 1M |
|---|---|---|---|---|
| Claude 3.5 Sonnet | $3.00 | $1.50 | $15.00 | $7.50 |
| GPT-4o | $2.50 | $1.25 | $10.00 | $5.00 |
| Claude 3.5 Haiku | $0.80 | $0.40 | $4.00 | $2.00 |
| Gemini 1.5 Pro | $1.25 | $0.625 | $5.00 | $2.50 |
3. The JSONL Payload Anatomy
Batch requests are submitted as JSONL files where each line encapsulates an independent HTTP request with a unique custom_id. This custom ID is your tracking key to correlate the returned response back to your database record.
{"custom_id": "eval-task-001", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "gpt-4o", "messages": [{"role": "system", "content": "You are a code evaluator."}, {"role": "user", "content": "Analyze time complexity of quicksort."}]}}
{"custom_id": "eval-task-002", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "gpt-4o", "messages": [{"role": "system", "content": "You are a code evaluator."}, {"role": "user", "content": "Analyze time complexity of mergesort."}]}}4. The 4-Step Production Lifecycle
1Upload the Payload File
Upload your .jsonl file to the provider's Files API with purpose set to "batch". The API returns a unique file_id.
2Create the Batch Job
Send a POST request to /v1/batches referencing the input_file_id and setting completion window to "24h".
3Poll or Listen for Webhook Notification
Periodically poll status (validating → in_progress → completed) or configure webhook listeners when the job transitions to completed.
4Stream & Ingest Results
Download the output_file_id. Stream each line, match the custom_id against your database, and inspect any failures in the separate error_file_id.
5. When to Use (and Avoid) Batch APIs
Ideal Batch Use Cases
- Automated Evals: Nightly regression testing across 50,000 unit test cases or LLM-as-a-judge scores.
- Data Extraction: Bulk parsing 100,000 PDF invoices or medical claim documents.
- Synthetic Data: Generating millions of synthetic instruction pairs for fine-tuning.
- Embedding Corpi: Re-indexing entire enterprise knowledge bases for RAG systems.
When to Stick with Real-Time
- Interactive Chatbots: User-facing web apps where latency must remain <1.5 seconds.
- Autonomous Agents with Multi-Turn Loops: Where turn N depends on step N-1 output immediately.
- Real-time Fraud/Security Screening: Where transactions must approve or decline synchronously.