Tokenization Explained: BPE vs WordPiece vs SentencePiece

1. The Problem With Character-Level and Word-Level Processing

Neural language models do not process text strings directly; they compute over vectors of floating-point embeddings indexed by discrete integers. When computer scientists first designed text models, two naive approaches failed:

• Word-level tokenization: Assigning an ID to every unique English word creates an intractable vocabulary (millions of entries), cannot handle typos (teh instead of the), and fails completely on out-of-vocabulary (OOV) terms.
• Character-level tokenization: Using single UTF-8 characters keeps the vocabulary tiny (256 bytes), but blows up sequence length by 4x to 5x. Because transformer self-attention scales quadratically or linearly with sequence length, processing character-by-character is prohibitively expensive.

2. How Byte-Pair Encoding (BPE) Solves the Dilemma

Originally a data compression algorithm published by Philip Gage in 1994, Byte-Pair Encoding was adapted for neural networks by Sennrich et al. in 2016.

BPE begins with a base vocabulary of individual bytes (256 items). It iteratively scans a multi-terabyte training corpus, finds the most frequently co-occurring pair of adjacent tokens, merges them into a new single token, and repeats this process until reaching a predetermined vocabulary budget (e.g. 100,000 or 200,000 tokens).

BPE Tokenization, Step by Step

Take a tiny training corpus with word counts: low ×5, lower ×2, newest ×6, widest ×3. Every word starts split into characters. BPE counts every adjacent pair, weighted by word frequency, and merges the winner:

After four merges, newest encodes as n · e · w · est and lower as low · e · r. A production tokenizer repeats this tens of thousands of times over terabytes of text. At inference, the learned merges are replayed in order on new text, so an unseen word like lowest still encodes cleanly as low · est.

Byte-Level BPE and Byte Fallback

Character-based BPE still needs an "unknown" token for characters it never saw. Byte-level BPE (introduced with GPT-2) runs the algorithm over raw UTF-8 bytes, so the 256-byte base vocabulary can encode any string — emoji, Cyrillic, or binary-looking garbage — with no unknown token. OpenAI's cl100k_base and o200k_base work this way. SentencePiece achieves the same guarantee with byte fallback: characters outside the vocabulary are emitted as their individual UTF-8 bytes. The cost is efficiency: a rare character can take 2–4 tokens, which is why non-Latin scripts cost more.

3. WordPiece Tokenization

WordPiece, developed at Google and popularized by BERT, builds its vocabulary the same bottom-up way as BPE but chooses merges differently. Instead of the most frequent pair, it merges the pair with the highest score:

This favors pairs whose parts rarely appear apart, i.e. merges that genuinely increase the likelihood of the training data. WordPiece marks pieces that continue a word with ##, so tokenization becomes token · ##ization. At inference it uses greedy longest-match-first instead of replaying merges. BERT-base uses a 30,522-token WordPiece vocabulary. WordPiece is common in encoder and embedding models but rare in today's generative LLMs.

4. SentencePiece and the Unigram Model

SentencePiece (Kudo & Richardson, 2018) is a tokenizer library, not a single algorithm. Its key idea is to skip language-specific pre-tokenization: it reads raw text as a stream of Unicode characters and treats the space as an ordinary symbol, written ▁. That makes it work equally well for languages without spaces, such as Japanese and Chinese, and makes decoding perfectly reversible.

SentencePiece can train two kinds of models: BPE (used by Llama 1 and 2 with a 32,000-token vocabulary) and Unigram. The Unigram language model (Kudo, 2018) works top-down: it starts from a large set of candidate substrings, assigns each a probability, and repeatedly prunes the tokens whose removal reduces corpus likelihood the least. When encoding, it picks the most probable segmentation, and during training it can sample alternatives ("subword regularization") to make models more robust. Google's Gemini and Gemma models use SentencePiece tokenizers with vocabularies around 256,000 tokens.

BPE vs WordPiece vs Unigram at a Glance

5. Frontier Tokenizer Architectures Compared

Today, different frontier AI labs utilize specialized tokenizer implementations tailored to their training data and target languages:

6. Why Tokenization Matters for Your API Bill

Because AI providers charge per 1M tokens rather than per word or character, tokenizer efficiency directly dictates cost. If Model A uses 1.1 tokens per word on your multilingual codebase while Model B requires 1.8 tokens per word, Model B effectively charges a 63% hidden premium even if their posted per-token rates appear identical.

See how many words fit in a token budget in the tokens to words guide, compare real counts across tokenizers in the tokenizer benchmark, or count your own text with the GPT, Claude and Gemini token counters.