The logit lens
Go deeper · Chapter 10, LM head — take the one matmul this chapter just built and slide it down the stack: read the vocabulary out of every depth, not only the last.
The LM head reads the next-token distribution out of the final hidden state by multiplying it through the tied unembedding W_U. Nothing in that matmul actually requires the hidden state to be the last one. So here is the one-line question that opened a whole strand of interpretability: what if you aim the same W_U read at a middle layer? That is the logit lens — coined by nostalgebraist in 2020 (“interpreting GPT: the logit lens”, LessWrong).
The idea: one read, every depth
A decoder LLM carries a single residual vector h_ℓ per token, refined boundary by boundary up the stack. The LM head only ever unembeds the top of that stack. The logit lens does the exact same thing at every boundary in between: take h_ℓ, push it straight through W_U, softmax, and read the top token — a crude “what would the model say if it had to answer right now?” at each depth. Line those reads up by layer and you can watch a prediction assemble itself.
See it live
The widget below runs precisely that on the Qwen3.5-0.8B loaded in your browser: one forward pass, then the tied W_U read at a dozen residual boundaries. Pick a short prompt and run it. Watch the per-layer top token climb out of gibberish in the middle of the stack into the real answer near the top — and watch a pinned answer token (say Paris for “The capital of France is”) climb the full-vocabulary ranks only in the late layers.
Read the model’s vocabulary straight out of a middle residual layer and you mostly get gibberish; only the late layers surface the real answer. Run it live on Qwen3.5-0.8B and watch the per-layer top token resolve with depth.
Interactive demo — loads after the page boots.
Two things are worth noticing. The middle rows are mostly noise — the model has not “decided” yet, so the top token there is often unrelated to the answer. And the answer does not fade in smoothly; it tends to snap into place in the last handful of layers. That is a recurring logit-lens observation: on this model the vocabulary-facing decision is made late, and the earlier layers are still doing something the unembedding cannot read cleanly.
Why it is only approximate — the blind spot
The logit lens is a rough probe, not ground truth, and it is worth being honest about why. W_U was trained to read exactly one distribution: the final layer’s. A middle h_ℓ lives in a different region of the representation space — it is off-distribution for the unembedding — so reading it through W_U is a shortcut that can be badly wrong, and it is worst in the early layers. On a shallow 0.8B model the interpretable band is thin and lands late: those noisy mid-stack reads are the method’s honest signature, not a defect in this demo.
The fix: a fitted lens (and what this page does not claim)
The blind spot has a fix: instead of the raw W_U, transport each h_ℓ through a small learned map first, so the read matches what that particular layer actually encodes. A tuned lens learns a per-layer affine probe (Belrose et al., EleutherAI, 2023); a Jacobian lens reads each h_ℓ through the network’s own averaged Jacobian. That Jacobian lens is Anthropic’s, developed and evaluated on their models, and it sharpens exactly the blurry middle layers you just watched refuse to resolve.
To be clear about scope: this page runs the plain logit lens only. It does not run a Jacobian or tuned lens, and it makes no claim to reproduce Anthropic’s results — it simply shows you, live, the blind spot that a fitted lens is built to close.