Thinking Machines dropped RoPE — idlemachines

← essays<br>← essays

Intro

Thinking Machines released their first model, Inkling, a 975B parameter MoE, with 41B of those active, and a 1M token context length.<br>It’s fair to say it’s the strongest open-weight model from a US lab so far, albeit a step behind the best Chinese models on agentic work. But it should be commended for strong omni-modal performance, including a rare foray into audio.<br>The architecture is mostly familiar, it wasn’t exactly the wholesale rethink that some commentators had been expecting, but more a thorough example of best practice.<br>There was one exception, however, Inkling does not use RoPE. And not in the way some other labs have done with substitutes, they do have positional encodings, but they introduce them via a combination of learnt local attention bias and a positionless far field.

The limited range of the attention bias means long context pairs carry no positional information at all, essentially becoming a content-only attention score outside the learnt bias. But because the bias is learnt at short range it gives a unique positional pattern for queries to keys in the window.

This is more than just an alternative way to reference position, they get at the uniqueness of how individual tokens relate to others in the sequence, how this is both content and context dependent. This is a fundamental shift in how positional information is thought of, and brings to bear the Bitter Lesson once again. The data knows these patterns better than you do.

note<br>This essay focuses on the positional encoding implemented in Inkling, but it has lots of other details worth checking out as well. The Inkling trending entry covers the model itself, with a full question set.

Where to inject position

Attention can be thought of as a set operation, where if we permute the order of the keys and values the output permutes with them. i.e. nothing about the product q⋅kq \cdot kq⋅k knows which token came first.<br>So, every transformer has to inject position somewhere, and broadly we can categorise most methods into three broad buckets.

Where position is incorporated: added to the input, rotated into q and k, or added to the logits

practice<br>Learned Absolute Position EmbeddingsEasyQ430<br>Learned Absolute Position Embeddings: the GPT-2 lookup table, and the limit it puts on context.

Option 1: add positional information into the input vectors<br>This is very much the original recipe, take the input token embeddings and concatenate the absolute or learnt position embeddings on top of the tokens. This basically makes every token

x′=xPE+xWEx' = x_{PE} + x_{WE}x′=xPE​+xWE​

xPE,xWE∈Rseq×dx_{PE}, x_{WE} \in \mathbb{R}^{\textrm{seq} \times d}xPE​,xWE​∈Rseq×d

practice<br>Sinusoidal Positional EncodingMediumQ187<br>Sinusoidal Positional Encoding: fixed frequencies, computable at any position.

This is simple to implement but as should be clear looking at the shapes above, as soon as the sequence extends beyond the training window the sequence position loses meaning. It’s not even obvious how to extend this to a sliding window, does each window step need all tokens to be re-embedded with position? How do you efficiently use this in a KV-cache?<br>These are just some of the reasons the move was broadly made to relative position encodings.

practice<br>Rotary Position Embedding (RoPE)HardQ191<br>Rotary Position Embedding: rotate q and k so relative position falls out of the dot product.

Option 2: Relative embeddings<br>The logic with relative embeddings is that the thing we care about in a sequence is how far apart two tokens are, that is what informs their meaning, and if these are at the start of a sequence or a million tokens into a stream, the same relationship should hold. It is a natural fit to streaming long context data, sliding windows, and the way this is implemented in RoPE through the properties of rotation matrices is a lovely clean bit of maths.<br>The core idea is that in RoPE we rotate each query and key vector by an angle determined by the position in the sequence across a spectrum of frequencies, but once we see these rotations in the dot product only the difference is required. (mechanics in the RoPE essay).

q⊤R(−mθ) R(nθ) k=q⊤R((n−m)θ) kq^\top R(-m\theta)\, R(n\theta)\, k = q^\top R\bigl((n - m)\theta\bigr)\, kq⊤R(−mθ)R(nθ)k=q⊤R((n−m)θ)k

As is clear in the equation, the dot product of two rotated vectors depends only on the difference of positions not the absolute positions, so relative position falls out of absolute rotations. Even better, the keys can be cached once, and it doesn’t need any trainable parameters.

The problem is that while the model performs well at short to medium ranges, the simplicity of the maths doesn’t translate into simple patterns being learnt at longer ranges.

Option 3: position as a bias to the attention

practice<br>ALiBi Attention BiasMediumQ194<br>ALiBi Attention Bias: the fixed linear penalty that Inkling's learnt bias descends from.

The idea is...