The Jacobian lens

Go deeper · Chapter 10, LM head — the fix for the blurry middle layers the logit lens could not read. And unlike that page, this one runs a fitted lens live on your in-browser Qwen3.5-0.8B.

The logit lens ended on an honest blind spot: a middle h_ℓ is off-distribution for the tied unembedding W_U, so pushing it straight through W_U gives noise in the mid-stack — exactly where you most want to read. The fix is to stop reading h_ℓ raw and first transport it into the frame the unembedding was trained for. That is the Jacobian lens, and unlike the logit-lens page, this one runs the fitted lens live: toggle it on and watch the same layers that refused to resolve become legible.

What a Jacobian lens is

Keep the LM head’s one matmul, but slip a per-layer map J_ℓ in front of it:

J_ℓ is a per-layer, corpus-averaged first-order (Jacobian) map of how a small nudge at layer ℓ moves the final residual. Applying it transports h_ℓ into the final layer’s frame before the tied unembedding reads it — so the read matches what the top of the stack expects, not what layer ℓ raw happens to look like. Two neighbours put it in context: the logit lens is this formula with $J_{\ell }=I$ (read the residual raw); a tuned lens replaces J_ℓ with a per-layer affine probe learned end-to-end (Belrose et al., EleutherAI, 2023).

Not a tuned lens — a different target

It is worth being precise about why this is not just a tuned lens with extra steps. A tuned lens is optimized so its readout agrees with the next token — it minimizes KL to the model’s own output distribution. The Jacobian lens optimizes nothing of the sort: it reads what a layer actually encodes by transporting it through the network’s own averaged Jacobian. Best next-token agreement and best intermediate readout are different goals, and when they disagree that is a feature, not a defect — the two lenses are answering two different questions.

What we fit — and what we did not

Be blunt about provenance. We fit the 23 maps J_1..J_23 on 100 WikiText prompts (wikitext-103-raw-v1, raw text, no chat template) on this exact Qwen3.5-0.8B — about 11 hours on a laptop — with the fit target set to the final residual taken before the final norm. The output boundary ℓ24 is left as $J=I$ by construction, so the fitted lens only ever acts mid-stack. The method is Anthropic’s; the fit and every number on this page are ours, on a small open model. The fit script and the vendored evaluation suites live in the repo (packages/browser/scripts/jlens/), Apache-2.0, so the whole thing is reproducible end to end.

See it live

Toggle LOGIT | JACOBIAN on a curated prompt. The baked frame renders instantly with no download; “compute live on your device” recomputes it on the in-browser model (and, for JACOBIAN, loads the fitted pack once). Watch a concept the logit lens keeps at rank 999+ snap into the top handful of ranks in the middle of the stack under the fitted J.

Does the fitted lens actually help?

A live demo can cherry-pick. So here is the aggregate, on our six vendored evaluation suites (the paper’s §methods-comparison prompts). The headline metric is a normalized log-k pass@k AUC — higher means the target/intermediate becomes legible earlier and at more depths — with the rank taken as the min over the fitted domain ℓ1..23. On this run the Jacobian lens beats the logit lens on 6 of 6 suites:

Read this honestly. A higher AUC does not mean “the model got it right” — it means the answer or an intermediate concept becomes readable earlier and across more layers under the fitted lens. And the two hardest suites, poetry and association, sit near the floor for both lenses (J-AUC 0.039 and 0.053; logit-AUC 0.031 and 0.003): the Jacobian lens still wins, but on tasks this hard for a 0.8B, neither lens reads much. Reproduce the whole table with JLENS_PACK=lens-pack-v1.safetensors JLENS_OUT=eval-results-v1.json oxnode packages/browser/scripts/jlens/eval.mts.

Where the gain lives

The advantage is not spread evenly up the stack — it is a mid-stack band. Four proxy detectors of band structure all fire (4/4), and the one that measures it directly — the fraction of evaluation intermediates where the J-lens rank beats the logit rank at a given boundary — rises out of the early band around ℓ6-7 and peaks at ℓ17 (fraction 0.595 vs an early-band fraction of 0.158). Note what the headline AUC is and is not: it is a min over the whole fitted domain ℓ1..23, not a band-restricted score — the band story explains where the min tends to come from, it does not change how the number is computed.

The French headline prompt in the widget is the qualitative face of this: for La saison après l’été est l’, the abstract concepts season and summer surface near ranks 1–2 around boundaries 16–17 under the fitted J, while the plain logit lens keeps them pinned at rank 999+ across the same depths.

What Anthropic saw at frontier scale

On their own models (Claude), Anthropic report the Jacobian lens surfacing rich, verbalizable intermediate content — with a workspace band roughly k~25 layers wide and readouts stable to within ≤10% variance. Those figures are Sonnet-4.5 numbers, on Anthropic’s models, in their paper Verbalizable Representations Form a Global Workspace in Language Models (transformer-circuits.pub, 2026). We neither run nor reproduce their causal, swap, or tuned-lens experiments. Whether a 0.8B model is anywhere near as rich is explicitly unknown — our 6/6 above is a readout-only partial reproduction on a small open model, not a claim about theirs.