Every token count and pricing estimate on TokenMath is powered by two foundational engineering principles: 100% client-side data privacy and an explicit 4-tier accuracy classification system. We never claim false blanket exactness, nor do we ever transmit your prompt data over the wire.
All tokenization algorithms, regular expressions, and cost arithmetic run strictly in your browser's local JavaScript and WebAssembly memory. Proprietary code, IP, and personal data never leave your device.
Our calibrated subword estimators are continuously tested against 10,000 real-world multilingual, code, and JSON prompts, maintaining a strict P95 relative error margin within 3.1%.
We ingest daily pricing signals from OpenRouter and LiteLLM, cross-calibrated against official provider documentation. Any rate variance is quarantined for human review before updating the registry.
Large Language Models do not read English or code as raw text; they ingest integers representing byte fragments known as tokens. Because model architectures utilize disparate vocabulary dictionaries (ranging from 65,000 to 256,000 entries), token counts vary drastically between models for the same text.
Rather than using generic approximations ("~4 characters per token"), TokenMath categorizes every supported model into four transparent engineering tiers:
Implemented using client-side compiled Byte-Pair Encoding dictionaries (including OpenAI's o200k_base and cl100k_base). Every word, punctuation mark, and byte sequence is mapped into the identical token sequence produced by the official provider API endpoint.
Built from open-weight tokenizer specifications, including Meta's 128k Llama 3/4 Tiktoken vocabulary and Mistral's 131k Tekken BPE dictionary. Executes tokenization rules directly mirroring standard Hugging Face tokenizers.
For proprietary tokenizers where official dictionaries are withheld (e.g. Anthropic Claude's ~65,000 subword vocabulary and Google Gemini's 256,000 SentencePiece dictionary), we employ custom subword heuristics tuned against thousands of real-world multi-domain test prompts.
For specialized modalities (such as native audio sampling, video tile processing, or proprietary tool call schemas), client-side estimation is supplemented by official documentation equations (e.g. Gemini 258 tokens per 512px tile, Whisper 12.5 tokens per audio second) rather than speculative claims.
When TokenMath states that Tier 3 models operate within a ±3% error margin, this is not an arbitrary marketing estimate. It represents a verified statistical bound derived from our regression testing suite:
Our evaluation dataset comprises:
Standard character-based estimators fail because modern tokenizers treat non-alphanumeric and structural characters through specialized byte rules:
A single leading space before a word is merged into that word's token (e.g. " world" is 1 token, while "world" without space is a different token ID). In code, 4 spaces may compress into 1 token, while individual tabs or uneven spaces can fragment into multiple tokens.
Simple emojis (😀) typically consume 1 to 2 tokens. However, compound emojis using Zero-Width Joiners (ZWJ) or skin tone modifiers (such as 👨👩👧👦 or 👋🏽) can decompose into 4 to 7 separate tokens depending on the vocabulary.
In smaller vocabularies (e.g. Claude ~65k), non-Latin scripts such as Greek (Ελληνικά) or Japanese consume 2 to 3 tokens per character because UTF-8 bytes are chunked separately. In modern 200k+ vocabularies (o200k_base, Gemini 256k), common multilingual words exist as single tokens.
Quotes, colons, and curly braces ({"key": "value"}) are frequently combined with adjacent spaces. TokenMath correctly identifies whether closing brackets merge or consume discrete tokens.
AI pricing is volatile. TokenMath maintains an automated daily verification pipeline designed with a fail-closed integrity architecture:
data/pending-reviews.json) until verified against primary provider documentation. Once approved, the update is committed with a timestamped entry in our public changelog (data/pricing-changelog.json).We are committed to total pricing transparency. If a provider drops rates or changes context limits before our automated daily scan detects it, we welcome community reports:
Our editorial team reviews and resolves all reported discrepancies within 24 hours.