Search papers, labs, and topics across Lattice.
This study explores how the grammatical roles of tokens influence the geometry of their representations in transformer models, specifically examining the evolution of Intrinsic Dimensionality (ID) across layers. The authors find that closed-class items expand and contract in dimensionality more rapidly than open-class items, which is linked to changes in neighborhood structures among tokens. By comparing various encoder and decoder architectures, the research reveals distinct patterns in how these models integrate contextual information, ultimately demonstrating that geometric features can effectively recover a token's grammatical role in downstream tasks.
Closed-class tokens expand and collapse in representation space faster than open-class tokens, revealing critical insights into how grammar shapes transformer geometry.
Transformer representations describe trajectories through high-dimensional vector spaces, which are shaped dynamically as tokens incorporate relational context across layers. Such data tend to concentrate on lower-dimensional sub-manifolds, a form of compression quantified by the Intrinsic Dimensionality (ID), the minimum number of independent variables needed to represent them without significant information loss. In this work, we ask whether the grammatical role of tokens, as marked by their part-of-speech (PoS) tag, shapes the local geometry of this manifold. To this end: (1) We investigate the layer-wise evolution of ID, finding that closed-class items expand earlier and collapse sooner than open-class ones; (2) We show its expansion and contraction to be explained by changes in the neighborhood structure, and hence in the relations between words within a sentence; (3) We compare encoders (ModernBERT, bigbird-roberta-large) and decoders (gemma-2-2B, Llama-3.2-3B), finding that the two families evolve differently across layers, consistently with how each integrates context;(4) We show that geometric features alone recover a token's grammatical role, and use them to interpret how the semantic content of each PoS evolves across layers in a downstream classification task.
