Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces Lexical Reference Frames (LexFrames) to interpret language model residual streams by approximating the native semantic reference frame induced by subsequent layers, rather than relying on fixed lenses or point cloud methods. LexFrames achieve near-perfect coherence (0.97) and read states within 0.16 nats of the model's output, significantly outperforming fixed lenses which are limited by the square root of native frame coherence. This approach enables states to be readable, attributable, and steerable, allowing for faithful prediction attribution and potential early layer exits.
Language model states are not static points but possess dynamic, context-adaptive reference frames that can be approximated to achieve near-perfect interpretability and steerability.
The vocabulary of a language model is its native semantics: the embedding and the unembedding fix the meaning of a state at the two ends of the residual stream. Between them, existing methods either reuse a fixed lens or treat the states as a point cloud, and neither reads a state the way the model does. We argue that every hidden state has a native reference frame, induced by the layers that read it next and adaptive to the context. Lexical reference frames approximate it by reading a state through the model's own remaining layers with a stated number of rounds of attention between positions and a typical attention output for the rest, without fitting to the model's output. No lens, with fixed directions per layer, can exceed the square root of the native frame's coherence, 0.30 and 0.35 at mid-depth of SmolLM3-3B and OLMo3-7B, whereas the first-order frame, LexFrame[1], reaches 0.97 and 0.97 and reads a state within 0.16 nats of the model's output. Its coefficients attribute predictions more faithfully than attribution patching, its lexical basis steers a prediction while sparing other tokens, and its confident readings would let an early exit skip 0.45 and 0.42 of the layers, where lenses skip almost none. In this way, lexical reference frames make every hidden state of the model readable, attributable and steerable.