Every token count and pricing estimate on TokenMath is powered by two foundational engineering principles: local client-side computation and an explicit 4-tier accuracy classification system. Prompt and calculator input data are processed strictly in local browser memory. No prompt text, system instructions, or workload parameters are transmitted across external networks.
Prompt text and calculator inputs are processed locally and are not transmitted to TokenMath servers. All tokenization algorithms, regular expressions, and cost arithmetic run strictly in your browser's local JavaScript and WebAssembly memory. Prompts and calculator inputs never leave your browser.
Our calibrated subword estimators are tested against 10,000 representative benchmark prompts. Empirical verification confirms:
We use exact raw tokenizer encoding wherever the provider publishes a canonical tokenizer, and explicitly separate raw token counts from provider-injected API delimiter framing tokens.
We ingest daily pricing signals from OpenRouter and LiteLLM, cross-calibrated against official provider documentation. Any rate variance is quarantined for human review before updating the registry.
Large Language Models do not read English or code as raw text; they ingest integers representing byte fragments known as tokens. Because model architectures utilize disparate vocabulary dictionaries (ranging from 65,000 to 256,000 entries), token counts vary drastically between models for the same text.
Rather than using generic approximations ("~4 characters per token"), TokenMath categorizes every supported model into four transparent engineering tiers:
Implemented using client-side compiled Byte-Pair Encoding dictionaries (including OpenAI's o200k_base and cl100k_base). Raw string tokens match official model vocabularies exactly. Minor divergence on API billing endpoints (typically 1 to 4 tokens) arises strictly from provider-injected ChatML message framing envelopes and assistant priming tokens. See our empirical Tokenizer Accuracy Benchmark & Delimiter Analysis.
Built from open-weight tokenizer specifications, including Meta's 128k Llama 3/4 Tiktoken vocabulary and Mistral's 131k Tekken BPE dictionary. Executes tokenization rules directly mirroring standard Hugging Face tokenizers.
For proprietary tokenizers where official dictionaries are withheld (e.g. Anthropic Claude's ~65,000 subword vocabulary and Google Gemini's 256,000 SentencePiece dictionary), we employ custom subword heuristics tuned against thousands of real-world multi-domain test prompts.
For specialized modalities (such as native audio sampling, video tile processing, or proprietary tool call schemas), client-side estimation is supplemented by official documentation equations (e.g. Gemini 258 tokens per 512px tile, Whisper 12.5 tokens per audio second) rather than speculative claims.
When TokenMath states that Tier 3 models operate within a ±3% error margin, this is not an arbitrary marketing estimate. It represents a verified statistical bound derived from our regression testing suite:
Our evaluation dataset comprises:
Standard character-based estimators fail because modern tokenizers treat non-alphanumeric and structural characters through specialized byte rules:
A single leading space before a word is merged into that word's token (e.g. " world" is 1 token, while "world" without space is a different token ID). In code, 4 spaces may compress into 1 token, while individual tabs or uneven spaces can fragment into multiple tokens.
Simple emojis (😀) typically consume 1 to 2 tokens. However, compound emojis using Zero-Width Joiners (ZWJ) or skin tone modifiers (such as 👨👩👧👦 or 👋🏽) can decompose into 4 to 7 separate tokens depending on the vocabulary.
In smaller vocabularies (e.g. Claude ~65k), non-Latin scripts such as Greek (Ελληνικά) or Japanese consume 2 to 3 tokens per character because UTF-8 bytes are chunked separately. In modern 200k+ vocabularies (o200k_base, Gemini 256k), common multilingual words exist as single tokens.
Quotes, colons, and curly braces ({"key": "value"}) are frequently combined with adjacent spaces. TokenMath correctly identifies whether closing brackets merge or consume discrete tokens.
AI pricing is volatile. TokenMath maintains an automated daily verification pipeline designed with a fail-closed integrity architecture:
data/pending-reviews.json) until verified against primary provider documentation. Once approved, the update is committed with a timestamped entry in our public changelog (data/pricing-changelog.json).We are committed to total pricing transparency. If a provider drops rates or changes context limits before our automated daily scan detects it, we welcome community reports:
Our editorial team reviews and resolves all reported discrepancies within 24 hours.