Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
The authors investigate how Qwen2.5-7B-Instruct executes number-comparison tasks to test whether transformer computation actively manipulates low-dimensional representational manifolds. They discover that despite the presence of curved manifold geometry, the model relies on linear representations by summing linear vector encodings into a shared residual stream space and using localized MLP neurons to compare bounded intervals. This establishes that low-dimensional manifold structure can passively coexist with purely linear computational circuits, challenging the assumption that geometric representation directly dictates algorithmic implementation.
Models don't need to manipulate curved manifolds to compute over them: Qwen executes number comparison via simple linear projections and localized MLP boundaries, largely ignoring the non-linear manifold geometry of its own representations.
One of the current premises of mechanistic interpretability research is that detailed accounts of the geometry of neural network representations can tell us how models perform computations, and how to effectively intervene on them. While low dimensional manifolds have been observed for multiple concepts in the literature (e.g. numbers encoded on helices, days of the week on a circle, ...), with structure believed to reflect properties of data and tasks, the extent to which models rely on them for computation, and how they manipulate them, remains unclear. We characterize precisely the geometry of computation in a number-comparison task, as an abstraction of comparison for decision making, and how models utilize geometry in an elegant fashion to implement it. Specifically, we study the causal geometry of number comparison in Qwen2.5-7B-Instruct, a capable and widely studied open-weight model, and find Qwen largely uses linear representations of numbers despite the presence of curved geometry. To compare two numbers, the model first encodes each number along a vector and adds the two representations using attention and the residual connection, bringing them into a shared space in the residual stream. Then, the model uses MLP neurons to compare the pair of numbers on local regions in this shared space, which correspond to smaller intervals of input numbers, and combines these to obtain the position of the maximum. In fact, this reliance on linear representations for comparison also persists when the model compares three numbers. Our findings demonstrate that the manifold hypothesis can co-exist with linear representations: while concepts that are ordered may have manifold structure in representations, the model may use an underlying linear structure of the concept in certain computations.