Search papers, labs, and topics across Lattice.
This paper empirically investigates relational linearity in language models, the hypothesis that the unembedding of an object can be predicted from the embedding of its subject via a linear map for a fixed relation. They introduce a KL-divergence-based probing method to evaluate relational linearity, avoiding Jacobian approximations used in prior work. Results across four datasets reveal that relational linearity varies across models and layers, and is sensitive to paraphrasing of relational queries.
Relational knowledge in LLMs isn't as simple as a fixed linear mapping between subject and object embeddings, and how you ask the question matters.
Linear properties are ubiquitous in the representations of language models; however, testing them experimentally remains a challenging task. This work focuses on relational linearity: the hypothesis that, for a fixed relation (e.g., "plays"), the unembedding of an object (e.g., "trumpet") can be predicted from the embedding of its subject (e.g.,"Miles Davis") by a linear map. We present an experimental method to test the formulation of relational linearity by Marconato et al. (2025). Specifically, we introduce a probing method, based on Kullback-Leibler divergence, to evaluate this property and examine its variation across layers and paraphrased relational queries. It is also more efficient than previous work; for example, it avoids the crude Jacobian approximations used in Linear Relational Embeddings by Hernandez et al. (2024). Our findings across four datasets show that relational linearity varies across models, exhibits layer-wise patterns consistent with prior observations about linguistic information in model representations, and is differently affected by changes in how the relation is phrased.
