Search papers, labs, and topics across Lattice.
This study investigates the geometric properties of pre-trained language model embeddings by analyzing the Riemannian structure of contextual token embeddings. The authors introduce Riemannian Mean Pooling (RMP), which utilizes pullback metrics from a learned encoder's Jacobian and aggregates them using the Fr茅chet mean on the SPD manifold, demonstrating that RMP outperforms traditional Euclidean mean pooling across three linguistically rich datasets. Notably, RMP maintains chance performance on a benchmark designed to eliminate lexical artifacts, indicating its robustness in distinguishing genuine linguistic signals from noise.
Riemannian Mean Pooling reveals that leveraging geometric properties of embeddings can significantly enhance classification performance while avoiding pitfalls of annotation-driven biases.
Understanding the geometric structure of pre-trained language model embeddings matters for interpretability and safety. We ask whether sentence-level classification signal lives in the Riemannian geometry of contextual token embeddings, and probe it by extracting per-token pullback metrics from a learned encoder's analytical Jacobian and aggregating them with the Fr\'echet mean on the symmetric positive definite (SPD) manifold; we call this procedure Riemannian Mean Pooling (RMP). Across three datasets with non-trivial linguistic structure (CoLA, CREAK, RTE), RMP outperforms Euclidean mean pooling, while on FEVER-Symmetric, a benchmark constructed to remove annotation-driven lexical artifacts, the method correctly stays at chance. Ablations show that a randomly initialised encoder combined with Fr\'echet aggregation already beats Euclidean pooling on two of the three signal-bearing datasets, localising the source of the gain to the geometric aggregation rather than to learned manifold structure; the trained encoder contributes additional signal specifically on CREAK, the most knowledge-heavy of the three signal-bearing datasets.
