Tokenization: turning text into the model's vocabulary

Before a transformer can do anything with your sentence, the sentence has to be cut into pieces the model knows. Those pieces are called tokens, and the rule that decides where the cuts go is called the tokenizer. Tokens are not characters, and they are not words — they are something in between, chosen by an algorithm called Byte-Pair Encoding (BPE). The widget on the right runs Qwen3.5's exact tokenizer in your browser. Type, watch the chips refresh.

Why sub-words?

Tokenizers sit in an awkward middle. If we made every word a token, the vocabulary would need millions of entries to cover the long tail of names, typos and made-up words — and the embedding matrix (the big lookup table holding one vector per vocabulary entry — next chapter) would balloon along with it. If we went the other way and made every character a token, the model would have to chew through sequences five-to-ten times longer than the same text in words, and every layer's cost is at least linear in sequence length.

One fixed sentence makes the trade-off concrete. Take "The quick brown fox jumps over the lazy dog": as characters it is 43 tokens; as whole words, 9; a sub-word tokenizer lands at exactly 9 — as short as words, but built from a vocabulary that never runs out of entries.

BPE finds a middle path: start from raw bytes (or characters), then repeatedly merge the most frequent adjacent pair into a new symbol until the vocabulary is the size you want. Common words like the end up as a single token; rarer words like Antidisestablishmentarianism decompose into shorter familiar pieces. The tokenizer doesn't know what a word means — it only knows what byte sequences are statistically frequent in the training corpus. Because that base alphabet is the 256 possible byte values, the tokenizer can represent literally any input — any Unicode character, emoji, or script, even ones it never saw in training — so there is no "unknown" or out-of-vocabulary token; unfamiliar text just breaks into more, smaller (byte) tokens.

What Qwen3.5 does specifically

Qwen3.5 ships with a BPE tokenizer of roughly 248,320 entries. A few quirks are worth knowing:

• Spaces live inside tokens. The BPE algorithm treats a space-prefixed word as a different symbol from the bare word, so "cat" at the start of a sentence and " cat" mid-sentence are usually two different tokens. The chips on the right mark a leading space with a small dot (·) so you can see this without it being invisible.
• Special tokens mark structure. Sequences like <|im_start|> and <|im_end|> aren't text the model generates — they're chat-turn boundaries used by Qwen3.5's chat template. The widget renders any token whose decoded text contains <|…|> with a dashed border so you can spot them.
• Multilingual coverage isn't uniform. Latin-script words are usually one or two tokens; CJK characters often land as one token each but unusual scripts can decompose into byte-level fragments. Try the multilingual example to see the difference.

Why this matters

Every downstream cost a language model has is paid per token: API billing, context-window limits (the cap on how many tokens the model can hold at once), KV-cache memory (the per-token working memory kept during generation — later chapter), and the wall-clock latency of generation. A prompt that looks short to a human can be long to the model if it uses rare punctuation, code, or languages the tokenizer didn't see much of in training. The characters-per-token ratio in the stats footer is the simplest way to feel this: prose in well-supported languages usually lands near 4 chars/token, code closer to 2-3, and an emoji-heavy or rare-script string can drop below 1.

Different model families use different tokenizers, and the same string can come out radically different lengths. That's why "tokens" is the lingua franca of LLM cost and capacity, not "characters" or "words."

Watching BPE build a token

The widget below walks through a stylised version of BPE on the input "unbelievable". Each step merges one adjacent pair of fragments into a new symbol — the same operation BPE training runs many millions of times against a corpus to learn its merge order.

Tokenization quirks worth seeing

Most surprises beginners hit with LLM cost or behaviour trace back to one of these. Each row shows a small input alongside the tokens it usually decomposes into.

In the next chapter (Embeddings) we'll see how each of these token ids becomes a high-dimensional vector that the model can actually compute with — the bridge between integers and the continuous space attention operates in.