Search papers, labs, and topics across Lattice.
This study investigates how large language models (LLMs) internally represent the concept of concreteness, particularly in the context of figurative language like metaphors. By conducting a layer-wise and geometric analysis across four model families, the authors reveal that LLMs effectively differentiate between literal and figurative uses of nouns in their early layers, while mid-to-late layers exhibit a consistent one-dimensional compression of concreteness. The findings not only illuminate the internal workings of LLMs but also demonstrate a practical application where a single direction in representation space can facilitate efficient classification and generation adjustments toward literal or figurative expressions.
LLMs can distinguish between literal and figurative meanings early in their processing, revealing a surprising geometric structure that simplifies figurative-language classification.
Static concreteness ratings are widely used in NLP, yet a word's concreteness can shift with context, especially in figurative language such as metaphor, where common concrete nouns can take abstract interpretations. While such shifts are evident from context, it remains unclear how LLMs understand concreteness internally. We conduct a layer-wise and geometric analysis of LLM hidden representations across four model families, examining how models distinguish literal vs figurative uses of the same noun and how concreteness is organized in representation space. We find that LLMs separate literal and figurative usage in early layers, and that mid-to-late layers compress concreteness into a one-dimensional direction that is consistent across models. Finally, we show that this geometric structure is practically useful: a single concreteness direction supports efficient figurative-language classification and enables training-free steering of generation toward more literal or more figurative rewrites.
