Multi-token prediction

Go deeper · Chapter 11, Sampling — how a model gets more than one token out of a single forward pass.

Sampling, from the last chapter, turns one row of logits into one token. To produce the next token you append it to the prompt and run the entire model again — all 24 layers, every weight. A 200-token reply is 200 full forward passes, strictly one after another, because each token depends on the one before it. That is the autoregressive tax, and it is the single biggest reason generation feels slow.

What makes it hurt is what each pass spends its time on. Producing one token touches every weight in the model exactly once, so a decode step is memory-bound: the GPU spends most of the step just reading the billion-ish parameters out of memory, and the actual math for that single token is a rounding error on top. The chip is mostly idle, waiting on memory. So here is the tempting question: if we are going to pay to stream all those weights in anyway, can we get more than one token for that price?

The bet: draft, then verify

Speculative decoding is the trick that says yes. The idea is to guess the next few tokens cheaply, then have the real model check all the guesses at once in a single forward pass. Checking k guesses costs about the same as producing one token normally — it is one pass over the weights — so every guess that survives is a token you got for free. Guess well and you emit several tokens per pass; guess badly and you have wasted a little cheap drafting, nothing more.

Classically you need a second, smaller “draft model” to do the guessing — an extra model to train, load, and keep in sync. Multi-token prediction (MTP) removes that cost: the model drafts for itself. Qwen3.5 ships one small extra module — the MTP head — whose only job is to propose the next token while the main model is still finishing the current one.

The MTP head Qwen3.5 carries

It is deliberately tiny, because it borrows almost everything from the main model. It takes two inputs: the main model's last hidden state for the current position, and the embedding of the token that was just emitted — and crucially that embedding comes from the same embedding matrix the main model uses (mtp_use_dedicated_embeddings: false). It normalizes both, fuses them with a single projection, runs them through one transformer layer, and reads out a token through the same tied LM head the main model uses. That is the whole head:

Two details matter for later. First, the head's one layer is a full-attention layer — not one of the 18 GatedDeltaNet layers — and it keeps its own little attention cache; it never touches the main model's recurrent state. Second, because the embedding and the output head are shared, the only new weights are three RMSNorms, one projection, and that single layer. In the config this is one line — mtp_num_hidden_layers: 1 — a single module that predicts one token beyond the next.

The loop: draft $D$, verify once, accept what matches

Now put the head to work. One decode step becomes three moves. Draft: the cheap MTP head proposes D candidate tokens, one after another (Qwen3.5's default depth is D = 1, though an engine can raise it). Verify: the full model takes the window [last committed token, d₁, …, d_D] and runs it through in one batched forward pass, getting its own opinion at every position at once. Accept: walk the draft left to right and keep each guess that matches what the full model would have picked; at the first mismatch, emit a correction instead and stop — at temperature 0 that correction is simply the full model's top token, while with sampling it is drawn from an adjusted distribution built from both models' probabilities, not literally the target's own pick. Step through it:

Count the payoff. If K of the D drafts are accepted, the step emits K + 1 tokens — the K good guesses plus one token the verify pass produced for free (the correction at the first miss, or a bonus token past the end if all drafts were right). So one pass over the weights yields anywhere from 1 token (everything rejected — the same single full-model pass as plain decoding, plus the small wasted cost of the draft) up to D + 1.

Why it is exactly lossless

Here is the part that makes speculative decoding honest rather than a quality shortcut: at temperature 0, a draft is kept only when it matches the full model's own top pick at that position. With sampling, the accept test is different but just as strict — it compares target and draft probabilities for that exact drafted token and accepts with probability $\min (1,p_{\text{target}}/p_{\text{draft}})$, resampling from a corrective residual distribution on reject. Either way the verify pass is the real model's real distribution; the MTP head never gets the last word. With the proper acceptance rule — the speculative sampling/decoding algorithm from Leviathan et al. and, independently, Chen et al., both 2023 — the tokens you emit follow the exact same distribution as plain one-at-a-time decoding — the same probabilities at every step and the same temperature behaviour, just produced in fewer passes (at temperature 0 that means the identical text; with sampling, the identical odds). The MTP head never changes the answer — at worst it costs a small drafting tax, at best it saves several passes.

That is also why a wrong guess is cheap: a rejected draft costs only the little bit of MTP compute that produced it. The expensive verify pass was going to happen regardless — it is the same forward pass plain decoding would have run to produce that one token. Speculative decoding is, at worst, plain decoding with a small drafting tax; at best, a multiple of the speed.

Where the idea came from

MTP did not arrive fully formed. Three steps got us here, and the design Qwen3.5 ships is the third:

Meta's 2024 version bolted four parallel heads onto a shared trunk, each predicting a different future token at the same time — great as a training signal, but the heads are independent, so they cannot condition on each other. DeepSeek-V3 changed the shape to a single sequential module that predicts one extra token while keeping the causal chain — the draft of token t+2 gets to see the chosen t+1. Qwen3.5 (built on the Qwen3-Next architecture) adopts exactly that sequential, weight-sharing design: one module, depth one.

It earns its keep twice

The same head pays off at two completely different times. During training, asking each position to predict the next two tokens instead of one is a denser learning signal — more to learn from every token of data. Meta reported their multi-token models solving 12% more HumanEval and 17% more MBPP problems at 13B; DeepSeek-V3 folds an MTP loss into pretraining with a weight that starts at $\lambda =0.3$ and drops to $\lambda =0.1$ for the final stretch of tokens:

Qwen describes its own MTP as boosting both pretraining efficiency and inference speed. During inference, the very same weights become the free self-drafter from the loop above. One module, two jobs: a better-trained model, and a faster one.

Where the time goes

Earlier we said a decode step is memory-bound — the chip mostly idle, waiting for weights. Speculative decoding only makes sense once you see that. Below is the story the way a GPU profiler would draw it: two timelines on one clock, plain decoding on top, draft → verify → accept underneath. Each pass is ≈20 ms wide because that is roughly what this tab's own plain decode measures (~50 tok/s on an M-series Mac) — and the verify pass is just as wide as a plain one, because either way it is one pass over the weights. Raise the draft depth, make the text harder to guess, and watch what happens to the trace:

Two things to notice. First, D has diminishing returns: draft number four only counts if drafts one through three all survived, so each extra guess is worth less than the one before it. Second, predictability gates everything — on hard-to-guess text the speculative lane sinks to scraps, barely ahead of plain decoding once the drafting tax is paid, and drafting deeper only grows the tax (watch D = 2 on surprising text fall behind D = 1). And look at what the red slots do to the timeline: nothing. Rejected positions never widen a pass — the waste rides a memory sweep you were paying for anyway. That is also why the trick pays at batch 1 and gets switched off on saturated servers: the free ride exists only while the compute row is empty.

A zoo of drafters

Everything above used the MTP head as the source of the cheap draft. But the draft → verify → accept skeleton doesn't care where the guess comes from — only that it's cheap, and that the verify pass keeps the lossless guarantee. Swap out the drafter and you get a whole family of speculative decoding, all sharing the exact same verify pass you just stepped through.

There's a draft-target setup (a second, smaller model writes the draft — easiest to bolt on, but you now run two models); Medusa (grow a few extra prediction heads on the model itself, no separate model); EAGLE (a purpose-built tiny model fed the target's hidden states — the go-to for general chat); and n-gram / lookahead decoding (no draft model at all — just look up what usually follows, which wins big on code and predictable syntax). The MTP head you just met is the one Qwen3.5 ships. Pick a drafter and compare what changes — and what doesn't:

Three rules hold across the whole zoo. They are all lossless — at temperature 0 verify keeps a draft only when it matches the real model's own top pick, and with sampling it runs the same probability-ratio accept/resample test from above, so every variant changes speed, never the answer. They all help most at low batch sizes, where the GPU has spare compute to run the verify pass for free; at high batch the chip is already saturated (the batching sub-chapter shows why), so servers turn speculation off. And higher temperature makes the next token harder to predict, which lowers the acceptance rate. Speculation buys speed when you're idle and the text is predictable, and quietly steps aside when it isn't.

Watch the real model verify — in one pass

Everything above was illustrative — canned tokens, scripted verdicts. This one is live: the actual Qwen3.5-0.8B you can load into this tab verifies a cheap draft in a single real forward pass, and every verdict below comes from the model's own logits. The draft comes from prompt-lookup — the n-gram drafter from the zoo above, really implemented here: the last few tokens of the prefix are looked up earlier in the prefix, and whatever followed there becomes the guess — or from your own typed continuation. It does not come from Qwen's MTP head; this checkpoint ships zero mtp.* tensors, as the next section explains. One more honesty note: the prefix is fed to the model as raw text, with no chat template — a raw language-model continuation, not a chat turn.

Three things to notice. The verify is greedy, temperature 0 — a draft token is accepted exactly when it is the model's own argmax at that position, nothing softer. On a reject, the model's own pick (the green correction chip) is what continues the text — that is why speculative decoding is lossless: the output is what plain decoding would have produced anyway. And the whole check — every position, every top-K distribution you can expand above — cost one forward pass over the weights. That single pass is the entire speed story this sub-chapter has been telling. (This widget is a one-off verify probe wired up for this demo — the browser's actual chat decode loop stays plain autoregressive, as the next section explains.)

Does your in-browser Qwen use it?

No — and, as usual, the honest answer is the interesting one. The architecture is real: the config for this exact model says mtp_num_hidden_layers: 1, so a full Qwen3.5 checkpoint does define the head you just saw. But two things keep it off the path in this demo. The bf16 checkpoint the browser downloads was converted without the MTP weights — of its 474 tensors, zero are mtp.* — and the WebGPU decode loop here is plain autoregressive, one token per pass, with no draft/verify machinery at all. (You are not alone: stock Hugging Face Transformers also discards the MTP weights; today it is mainly inference engines like vLLM, SGLang, and this project's own native Metal engine that actually run the loop — measured there at roughly 1.06–1.46× faster, depending on how predictable the text is.)

So multi-token prediction sits with the vision encoder and the other parts of the full checkpoint that the architecture defines but this browser tour does not walk: real in Qwen3.5, switched off here on purpose. You can even see it greyed out in the architecture map — the “MTP head” node, off the route a token actually travels in this demo.