What Is Byte Pair Encoding? How AI Splits Text

Byte pair encoding, or BPE, is a method for splitting text into tokens by repeatedly merging the most frequent pair of adjacent symbols in a training corpus. Common words end up as single tokens, rare words get broken into reusable pieces, and nothing is ever unknown because the pieces can always fall back to single characters.

That one idea sits underneath how many language models read text. It is why a model can handle a word it has never seen, and it is why your prompt costs more tokens than its word count suggests. The best way to understand it is to run it.

Where byte pair encoding comes from

BPE started as a data compression idea. The paper that brought it into language models, Neural Machine Translation of Rare Words with Subword Units by Sennrich, Haddow and Birch, first submitted in August 2015, examined segmentation methods "based on the byte pair encoding compression algorithm". The problem they were solving was rare words. A vocabulary of whole words cannot cover them all, so the authors split rare words into subword units the model already knows. They reported gains of 1.1 BLEU on English to German and 1.3 BLEU on English to Russian over a back-off dictionary baseline.

The algorithm in one paragraph

Start with every word split into single characters, plus an end-of-word marker. Count how often each adjacent pair appears across the corpus, weighted by word frequency. Merge the most frequent pair into a new symbol everywhere it occurs. Repeat until you have the vocabulary size you want. The ordered list of merges is the tokenizer.

A worked example you can check by hand

Take a tiny corpus with four words and their counts: low (5), lower (2), newest (6) and widest (3). Write </w> for the end-of-word marker. The merges below were produced by running the algorithm on exactly this corpus.

A few things to notice.

At step 1 three different pairs tie at a count of 9: e s, s t and t </w>. Real implementations break ties by a fixed rule, and the choice does change the exact vocabulary. The tie-break is part of the tokenizer.

After eight merges, the corpus reads as low </w>, low e r </w>, newest</w> and w i d est</w>. The word newest is now one token because it is frequent. The word widest is still split into w i d and est</w>, and that last piece is shared with newest. That sharing is the whole payoff: est</w> becomes a reusable suffix, so a word never seen in training, like "tallest", can still be written as pieces the model knows.

The word lower became low plus e and r. The model sees a familiar stem and two leftovers. Nothing here knows what "lower" means. It only knows which character sequences were common.

What real tokenizers add

Production tokenizers keep this core and change the details. Many operate on bytes rather than characters, so any text in any script can be represented, which is where the name comes from. Vocabularies run to tens of thousands of entries instead of our eight merges. Rules for splitting on spaces and punctuation come first, so merges do not cross word boundaries in unwanted ways. Check your model's own documentation for its specifics, because tokenizers differ between model families and are not interchangeable.

For the broader idea of what a token is and how counts translate into cost, what a tokenizer is is the companion read, and how to estimate tokens for an AI task turns it into a budgeting habit.

Three behaviors BPE explains

Spelling and counting letters are hard for models. The model receives newest</w> as one unit, not as six letters. Asking how many letters a word has requires knowledge about the inside of a token, which the model has to learn indirectly.

Leading spaces and capitalization change the tokens. In many tokenizers, " cat" with a space and "cat" without are different tokens, and "Cat" is different again. The same word can cost a different number of tokens depending on where it sits in a sentence.

Some languages and strings cost more. A tokenizer trained mostly on English has many merges for English and few for other scripts, so the same meaning can take more tokens elsewhere. The same goes for random strings, long identifiers and unusual formatting, which are rare in the corpus and break into many small pieces. This is a general tendency, and the size of the effect varies by tokenizer.

All three are consequences of one design choice: the model sees pieces, not letters or words.

Why it matters for building

Token counts drive your costs and your limits. A context window is measured in tokens, so text that tokenizes poorly fills it faster. Each token is then turned into a vector by an embedding, which is where the model's work starts. If you want the whole path from text to answer, how AI models work strings these pieces together.

FAQ

What does byte pair encoding do?

It builds a vocabulary of subword pieces by repeatedly merging the most frequent adjacent pair in a training corpus. Text is then split using those learned merges.

Is BPE the same as a tokenizer?

BPE is one method for training a tokenizer. A tokenizer is the full system that turns text into token IDs, and not all of them use BPE.

Why do AI models struggle to count letters?

Because they read tokens, not individual letters. A common word is often a single token, so its spelling is not directly visible to the model.

Does the same text always produce the same tokens?

With the same tokenizer, yes. Different models can use different tokenizers, so the same prompt can have different token counts across providers.