Search papers, labs, and topics across Lattice.
This paper introduces Semantic Reference Frames (SemRF), a formalism that disentangles semantic measurement from residual dynamics in language models by fixing embedding anchors and measuring states against them. By employing pseudo-inverse tying and establishing a semantic Voronoi diagram, the authors achieve stable semantic-basis coordinates and define a minimum-action path that reveals the depthwise semantic trajectory of computation. The findings indicate that lower trace complexity correlates with parameter efficiency, suggesting that optimizing for lower-action and lower-complexity traces can lead to more effective use of semantic degrees of freedom in language models.
Semantic Reference Frames reveal that optimizing the trajectory of language model computation can significantly enhance parameter efficiency and reduce complexity.
Residual-stream analysis asks how language-model computation evolves across depth, but intermediate decoding requires comparable readout coordinates across layers. If embedding anchors and unembedding readout disagree on the chosen span, apparent motion may reflect measurement drift rather than computation. We introduce \emph{Semantic Reference Frames} (SemRF), an anchor-based formalism separating semantic measurement from residual dynamics. A SemRF fixes anchors and measures states against them. Pseudo-inverse tying gives exact synchronization; under restricted bi-invertibility, SemRF yields stable semantic-basis coordinates, distortion bounds, and near-identity changes. With the frame fixed, residual computation becomes a depthwise semantic trajectory. The anchors induce a semantic Voronoi diagram: distance, or evidence such as logits, assigns each layer to a coarse cell, while coordinates retain within-cell motion and margins. We define layerwise steps, contribution profiles, and imbalance diagnostics, then use the Voronoi trace to define a margin-relaxed tube. The canonical trace is the minimum-action path inside this tube; when nonempty with positive quadratic weight, it is unique and obeys a discrete spline equation away from active constraints. Excess action controls step, curvature, and profile mismatch. Low curvature implies piecewise-linear compressibility and local knowledge density: lower trace complexity means fewer semantic knots. Through the parameter-to-trajectory map, this gives a conditional link to parameter efficiency: among admissible settings fitting data, lower-action and lower-complexity traces use fewer semantic degrees of freedom. The guarantees require controlled interface error and small projection residual under explicit tube constraints.
