Search papers, labs, and topics across Lattice.
The paper introduces DySem, a training-free framework for semantic textual similarity (STS) that identifies semantic-related internal components of LLMs using multilingual consensus. DySem constructs text-dependent joint semantic sets, enabling dynamic, sample-specific semantic dimensions for similarity calculation, rather than relying on static last-layer hidden states. Experiments across various LLMs demonstrate that DySem consistently outperforms recent baselines while using lower-dimensional representations.
LLMs can achieve better semantic similarity scores with lower-dimensional representations by dynamically selecting relevant semantic components, rather than relying on fixed, high-dimensional hidden states.
Calculating semantic textual similarity is a foundational task in natural language processing. Current large language models (LLMs) based methods typically rely on extracting last-layer hidden states with fixed dimensions to compute similarity for every text pairs. We argue that this paradigm is suffer from two limitations: (i) The last hidden layer encodes more general knowledge rather than just semantic knowledge, making it suboptimal for semantic similarity computation; (ii) The hidden layer dimensions of LLMs are generally very large, which introduces some redundancy and noise for representing semantics. In this work, we propose DySem, a novel training-free framework that investigates more semantic-related internal components of LLMs via multilingual consensus, and shifts away from static representation spaces in favor of dynamic, sample-specific semantic dimensions by constructing text-dependent joint semantic set and computes similarity over this shared dimensional subset. Extensive experiments across various LLMs show that our method consistently outperforms recent baselines while maintaining lower dimensions for similarity calculation. The code is released at https://github.com/szu-tera/DySem.
