Multimodal Vision Token Math: OpenAI 512px Tiles, Claude & Gemini
How do Vision Transformers (ViT) convert a 4K diagram, PDF page, or phone screenshot into integer token sequences? Learn the exact formulas OpenAI, Anthropic, and Google use to calculate and bill image and video tokens.
1. How Vision Models Ingest Images: Patches vs Words
Unlike text strings which are divided into subword character chunks by BPE or SentencePiece tokenizers, images are continuous 2D grids of RGB pixels. Vision Transformers divide images into fixed-size square patches (typically 14x14 or 16x16 pixels), project each patch through a linear embedding layer, and append learned 2D positional embeddings.
However, because image resolutions vary from 100x100 icons to 4000x3000 camera photos, AI API providers do not bill raw pixels. Instead, each provider employs an algorithmic preprocessing pipeline to resize, tile, and price input images.
2. OpenAI Vision Tiling Algorithm (GPT-6, GPT-5, o3)
OpenAI operates two distinct fidelity modes for image inputs: low detail and high detail.
- Max Resolution Constraint: If either dimension exceeds 2,048 pixels, downscale preserving aspect ratio so the longest side is 2,048px.
- Min Edge Scaling: Scale the image such that the shortest side is exactly 768 pixels.
- 512px Grid Tiling: Divide the scaled image into 512x512 pixel tiles. Any partial tile is counted as a full tile (
Math.ceil(w / 512) * Math.ceil(h / 512)). - Token Math: Each 512x512 tile costs exactly 170 tokens, plus a flat 85 base tokens for metadata framing.
Example (1920x1080 Full HD image):
1. Fits within 2048x2048 → no initial crop.
2. Shortest edge is 1080px. Scaling to 768px results in 1365x768 pixels.
3. Tiles across: ceil(1365 / 512) = 3. Tiles down: ceil(768 / 512) = 2. Total tiles = 6.
4. Total tokens: (6 * 170) + 85 = 1,105 tokens.
3. Anthropic Claude Vision Formula
Anthropic uses a continuous area-based compression formula rather than discrete tiles:
1. If longest edge > 1568px, downscale preserving aspect ratio so longest edge = 1568px.
2. Compute token count: tokens = Math.ceil((width * height) / 750).
3. Minimum floor: Claude images require at least ~80 tokens.
For the same 1920x1080 screenshot:
1. 1920px exceeds 1568px → downscaled to 1568x882 pixels.
2. Math.ceil((1568 * 882) / 750) = 1,844 tokens.
4. Google Gemini Vision & Video Tokens
Google Gemini approaches multimodal inputs with a standardized crop unit:
- Still Images: Gemini charges a flat 258 tokens per crop/image for standard resolution inputs.
- Video Streams: Gemini samples video frames at 1 frame per second (1 fps). Each video second consumes 258 visual tokens + 32 audio tokens = 290 tokens/second (17,400 tokens per video minute).
5. Production Cost Optimization Rules
In PDF extraction and screenshots, cropping empty white margins often reduces the tile grid from 6 tiles down to 2 or 4, slashing image token consumption by 33% to 66%.
If you only need to classify an image (e.g. "is this an invoice or receipt?"), set detail mode to low. A 4K image drops from 1,445 tokens to just 85 tokens.