What Is a Tokenizer? A Plain English Guide

A tokenizer is the component that converts your text into the numbered pieces a language model can actually process, and converts the model's output back into text. It is not part of the model. It is a separate lookup table plus a splitting algorithm, fixed at training time, and it decides how many tokens your prompt costs before the model sees a single one of them. That last point is why the same sentence, translated into a different language, can cost you two or three times more to send. This explains how the splitting works and what you can do about the bill.

The mechanical version

A model has a vocabulary: a fixed list of pieces, usually somewhere between 30,000 and 200,000 entries, each mapped to an integer. The tokenizer's job is to chop incoming text into pieces that appear in that list, and it is greedy about using the longest pieces available.

Most modern tokenizers use byte pair encoding. The vocabulary is built by starting from individual bytes and repeatedly merging the most frequent adjacent pair in a training corpus. Do that thousands of times and common English words end up as single entries, common prefixes and suffixes end up as their own entries, and everything else falls back to smaller fragments.

So `the` is one token because it appeared constantly during vocabulary training. A rare surname is likely three or four tokens, stitched together from fragments. A string of random characters is close to one token per character, because no merge ever learned that sequence. The model then works entirely in those integers. If you want the layer above this, we have a separate explainer on what a token in AI actually is.

Why identical meaning costs different amounts

Here is the part that surprises people who ship products in more than one language. Vocabulary training is dominated by whatever the corpus contained, which for the major commercial models means overwhelmingly English text. Merges that compress English got learned. Merges that would compress Thai or Amharic largely did not.

OpenAI publishes the English rule of thumb in its documentation on counting tokens: roughly one token per four characters, or about three quarters of a word. It also states plainly that other languages, code, numbers and emoji usually consume more tokens per word.

None of these are penalties anyone designed. They fall out of which merges the vocabulary happened to learn. But the effect on a product is real: a per-token price is not a per-meaning price, and a support chatbot serving customers in a non-Latin script can run several times the cost per conversation of the English version at identical usage.

What this explains that nothing else does

A few recurring confusions dissolve once you picture the vocabulary.

Models are bad at counting letters in a word because they never see letters. They see one integer for `strawberry`, not ten characters. Asking for a letter count is asking about information the tokenizer discarded before the model got involved.

Trailing whitespace changes output quality more than it should. A space before a word is usually part of the word's token, so a prompt ending in a stray space can push the model into an odd continuation. If a prompt behaves strangely and you cannot see why, check the end of the string.

Two models given the same prompt do not receive the same input. Different vocabularies mean different splits, which is one of several reasons a prompt tuned on one model degrades on another. That is worth remembering when testing a new model before switching.

Reducing the token bill without wrecking the prompt

The tempting move is to compress your prose. Strip articles, abbreviate, write in note form. It saves a little and costs a lot, because instruction-following degrades when instructions stop reading like language.

The savings are structural instead.

• Cut repeated context rather than words. A system prompt resent on every turn is the largest recurring line item in most applications, and prompt caching addresses it directly.

• Send identifiers, not payloads. Passing a 4,000 token document so the model can quote one clause is expensive. Retrieve the clause, send the clause.

• Watch your JSON. Deeply nested keys repeated across a hundred array entries can outweigh the values they describe. Flatter shapes and shorter key names cut real tokens with no loss of meaning.

• Measure in the target language, not in English. If you serve multiple locales, run your actual prompts through a token counter for each one before setting a price. Our guide to estimating tokens for an AI task covers the method.

That last point is where teams lose money quietly. Pricing modelled on English usage and deployed globally is a forecast built on the wrong unit.

FAQ

Is the tokenizer part of the model?

No. It is a separate component with its own vocabulary file, fixed when the model was trained. You can run a tokenizer on your own machine without the model to count tokens before sending anything.

Why do models struggle to count letters in a word?

Because they never receive letters. The tokenizer converts a word to one or more integers, and the character-level detail is gone before inference starts.

Do all models use the same tokenizer?

No, and this is why identical text produces different token counts across providers. Vocabularies differ in size and in which merges they learned, so a prompt priced accurately for one model may not be for another.

Does a bigger vocabulary make a model better?

Not directly. A larger vocabulary compresses text into fewer tokens, which helps cost and effective context length, but it also enlarges the embedding and output layers. It is an engineering tradeoff, not a quality dial.

Why does non-English text cost more?

The vocabulary was built mostly from English text, so the merges that compress English exist and the equivalents for many other scripts do not. The same meaning therefore splits into more pieces, and you pay per piece.