Search papers, labs, and topics across Lattice.
This paper introduces ReverseEOL, a method that enhances training-free text embeddings by utilizing reversed input text to provide additional context for decoder-only LLMs. By combining standard forward embeddings with reversed embeddings, the approach mitigates the limitations of causal attention, leading to more comprehensive contextual representations. Experimental results on STS and MTEB benchmarks show significant performance improvements over existing training-free methods across various LLM architectures and scales.
Reversing input text can unlock hidden contextual information, leading to richer embeddings and substantial performance gains in training-free LLMs.
Recent advances in Large Language Models (LLMs) have opened new avenues for generating training-free text embeddings. However, the causal attention in decoder-only LLMs prevents earlier tokens from attending to future context, leading to biased contextualized representations. In this work, we propose Reverse prompting with Explicit One-word Limitation (ReverseEOL), a simple yet effective method for enhancing the representational capability of frozen LLMs. ReverseEOL augments the standard forward embedding with an additional reversed embedding derived from the reversed input text. Since reversing the input exposes each token to context inaccessible in the original order, the resulting reversed embedding effectively provides complementary information to the original one. As a result, combining the forward and reversed embeddings yields a richer final representation. Comprehensive experiments on STS and MTEB benchmarks demonstrate that ReverseEOL significantly improves the performance of existing training-free baselines across a broad range of LLMs with diverse architectures and scales. Extensive ablations and analyses further confirm the necessity of our reversal mechanism.
