Positional encoding: how the model knows token order

Self-attention has a strange property: it is permutation-invariant. If you shuffle the input tokens, the attention output for each token shuffles right along with them, but the relationships the attention layer computes don't change. Without help, a transformer literally cannot tell "the cat sat on the mat" from "mat the on sat cat the". Something has to inject the order back in.

See it for yourself first. Below is a toy bag of three tokens with no position attached — reorder them and the attention weights never budge:

Two earlier ideas

• Learned positional embeddings. Early GPT, BERT, and friends gave every position 0..N-1 its own learned vector andadded it to the token embedding. Simple, but the model can't extrapolate to positions it never saw during training, and you have to pick a max length up front.
• Sinusoidal embeddings. The original 2017 transformer used fixed sinusoids of different frequencies, also added to the token embedding. Extrapolation is in principle possible, but in practice quality degrades past the training window.

RoPE's idea: rotate, don't add

Rotary Positional Embedding (RoPE) leaves the embeddings alone and instead bakes the position directly into the attention math. The trick is to rotate the query and key vectors by an angle that depends on the token's position,before the dot product is computed.

Concretely, RoPE pairs up adjacent feature dimensions — (x_0, x_1), (x_2, x_3), … — and treats each pair as a point in a 2D plane. For pair index i and rotary dimension d (the number of features RoPE actually rotates — not necessarily the full head dimension; more on that just below), the frequency is:

Reading that in words: as the pair index i grows, the exponent −2i/d gets more negative, so ${\theta }_i$ gets smaller. Low-index pairs spin fast (they track fine, local position); high-index pairs spin slowly (they track coarse, long-range position).

At token position m, the pair is rotated by an angle of $m\cdot {\theta }_i$:

A rotation matrix just spins a 2-D point — think of a clock hand — by an angle $\theta$; the four trig entries below are simply the recipe for computing where the spun point lands. The clock-hand dials further down show exactly this spin. Angles here are measured in radians — a unit where one full turn is $2\pi \approx 6.28$, so a rate of "1 rad/token" is about a sixth of a turn per step.

Run that matrix once by hand, on pair 0 — whose frequency works out to exactly θ₀ = base⁰ = 1 radian per token, so pair 0 spins 1 rad every step. Take the pair (1, 0) at position m = 2: it rotates by 2 rad, landing at (cos 2, sin 2) ≈ (−0.42, 0.91). Same length, new direction — and it is exactly the spin you will see on dial 0 (the high-frequency clock) further down.

The pairs are drawn here as neighbours (x_0 with x_1) for clarity; how Qwen3.5 actually lays them out is an implementation detail you can skip on a first read (see Advanced below).

Rotation is unitary — it preserves vector norms — so RoPE doesn't change how big Q and K are, only where they point. The position information rides on the angle, not the magnitude.

Wide range of frequencies

Low-index pairs (small i) get the highest frequencies and rotate fast — one full turn every few tokens. High-index pairs get vanishingly small frequencies and barely move even over tens of thousands of tokens. The intuition: high-frequency pairs encode fine local position ("am I 3 tokens from my neighbour?"), low-frequency pairs encode coarse global position ("am I in the early part of the document or the late part?").

Qwen3.5 uses a head dimension of 256, a partial rotary factor of 0.25 (so only the first 64 features per head are rotated — the rest pass through untouched), and an unusually large RoPE base of 1e7. The large base stretches the frequency spectrum so the lowest-frequency pairs barely rotate at all over the model's 262,144-token context window — a key ingredient for long-context extrapolation.

Why rotate only a quarter of the head? Because the unrotated 192 dims carry no position at all — they do pure content matching ("does this key mean what my query is asking for?"). Position only needs to ride on part of the vector; the rest is free to care about meaning alone.

One more wrinkle specific to Qwen3.5's hybrid design: this RoPE rotation is applied only on the 6 full-attention layers (those rotate the first 64 of 256 head dims, as above). The other 18 layers are linear (GatedDeltaNet) and encode position implicitly through their recurrence plus a short causal convolution — no RoPE at all.

The relative-position property

Here is the part that makes RoPE click. If you take the dot product of RoPE(q, m) with RoPE(k, n), the answer depends only on the difference m - n, not on m and n separately. The model never has to learn what "position 1,427" means — it only has to learn how attention should behave at relative offsets, which generalises naturally to lengths it has never seen.

The third panel on the right makes this concrete. Holding the query position fixed at m = 50 and varying the key position n, the rotated dot product peaks sharply at n - m = 0 and falls off (with ripples) either side as you move further apart. That fall-off is exactly the inductive bias that makes attention "want" nearby tokens more than distant ones, with no per-position parameters at all.

You can feel the offset-only dependence directly. Shift both positions together and the score never budges:

This chapter visualises RoPE's math directly — no model inference needed. Want to see how this plays out on real attention scores? Open Self-attention (Chapter 4) and look at how the score map favours the diagonal.