Sampling: turning logits into the next token

Every forward pass through Qwen3.5 ends the same way: a vector of logits — one real number per token in the model's vocabulary (~248k). A logit is an unbounded score: how strongly the model recommends that token as the next one. Sampling is the step that turns that vector into a single concrete choice.

Why not just pick the maximum?

The simplest rule is greedy decoding: take argmax(logits) at every step — argmax just means "pick the position with the largest value", i.e. the single highest-scoring token. It's deterministic and reproducible, but it has a famous failure mode — repetition. The instant the model finds a phrase whose continuation it's confident about, it keeps re-entering the same loop, because that loop is always the locally-best choice. Greedy decoding also throws away a lot of information: if two tokens have nearly equal logits, picking one and ignoring the other is a fragile tie-break.

Softmax: logits → probabilities

To sample, we first convert logits into a proper probability distribution with the softmax function:

Two things are happening here. The exponential makes every value positive; the division by the sum makes them sum to 1. Each logit is also divided by a temperature T before the exponential — the same T appears in every term, top and bottom. Temperature acts as a sharpness knob:

• T < 1 sharpens the distribution — high-logit tokens dominate even more, so output looks more like greedy.
• T > 1 flattens the distribution — small differences in logits are washed out, so output is more diverse but less coherent.
• T = 0 collapses to greedy (argmax). The widget clamps to a tiny positive value to avoid dividing by zero, which is numerically equivalent.

Why this exact function?

Softmax is really a smooth, differentiable stand-in for argmax. Where argmax snaps hard to the single largest logit, softmax slides between the candidates. Feed it [5.0, 4.9, 1.0] and you get roughly [0.52, 0.47, 0.01]: the two close leaders share the mass almost evenly while the third stays negligible. Widen the gap to [8.0, 4.9, 1.0] and the output sharpens to about [0.96, 0.04, 0.00] — nearly a hard argmax, which is exactly the limit greedy decoding (T = 0) walks toward.

That smoothness is also why softmax turns up far beyond sampling: because it has a clean gradient everywhere, it is the function the model is trained against, and it is the same normalization that weighs the keys inside attention.

To see why the distribution comes out the way it does, here is the computation broken into its four mechanical phases — raw logits, divide by T, exponentiate, normalize — looping one step at a time. Use the T buttons to run the same eight logits at 0.2, 0.7 and 2.0: the normalize-phase bars spike onto one token at 0.2 and flatten across all eight at 2.0.

Same seed, three temperatures

The bars above are the cause; here is the effect. We gave the real Qwen3.5-0.8B checkpoint — the same one this playground runs — the same seed text three times and only changed the temperature. At T = 0 the sampler is greedy: every step takes the single most probable token, so the output is identical on every rerun and tends toward the safest, blandest phrasing — notice how quickly it starts orbiting the same few words. At T = 0.7 the distribution is sharpened but not collapsed, and the sampled story — this run, at least — reads naturally; reruns differ.

The T = 1.5 run shows why high temperature degrades so fast: generation is a loop, and every sampled token is appended to the context that produces the next distribution. One improbable pick would be survivable — but at high temperature improbable picks pile up, each one dragging the context further from anything the model has seen — in this run the errors compound until the output is word salad.

Top-p (nucleus) sampling

Even with a sensible temperature, the long tail of the vocabulary still has tiny non-zero probability mass — and once in a while the sampler will land there. Most of those tail tokens are nonsense in context. Top-p sampling (also called nucleus sampling) trims the tail:

• Sort tokens by probability, descending.
• Walk the sorted list and keep accumulating probability until the cumulative sum reaches p.
• Throw everything after that away.
• Renormalize the survivors so they sum to 1 again.
• Sample from this nucleus: draw a random number r between 0 and 1, walk down the list adding up probabilities, and stop at the first token whose running total passes r — bigger slices get hit more often.

A common default is T = 0.7 with top_p = 0.9: the temperature gives the model some room to be creative, and top-p guarantees we never sample from the absurd tail. The widget on the right lets you sweep both knobs over a captured run and see what would have happened.

Slide the cutoff below and watch the nucleus form: tokens are kept top-down until their cumulative probability reaches p, the tail is discarded, and the survivors are renormalized so they sum to 1 again.

How the widget works

Pressing Run generates 6 tokens with the inspector capturing the top-16 raw logits at every step — the capture itself always runs at temperature=0, i.e. greedy. The temperature and top-p sliders then re-apply softmax + truncation to the cached logits — no re-running the model. Because dividing every logit by the same T can't change their rank order, and top-p can never drop the #1-ranked token, the highlighted bar is guaranteed to stay the tallest bar under every slider setting — it's always the model's greedy pick for that step. What actually changes as you move the sliders is the confidence gap to the runner-up bars: a wide gap means a real sampler would almost always agree with this greedy pick, a narrow gap means it would often disagree and land on a different token.

One caveat on the heights: the bars are softmax/top-p renormalized over only the captured top-16 logits, not the full 248,320-token vocabulary. The omitted tail still carries real probability mass, so every bar reads a little higher than its true full-vocab probability — most noticeably at high temperature, where the model spreads more mass into that discarded tail.

When sampling goes wrong

Two failure modes are worth seeing side-by-side: a low-temperature greedy run that loops, and a high-temperature run that turns into gibberish. The middle panel shows a sensible production setting.

Tip: try a low temperature (0.2) on a confident step versus a high temperature (1.5) on the same step. The visible "shape" of the bar chart is the entire reason sampling parameters matter for an LLM.