Search papers, labs, and topics across Lattice.
This paper introduces SHIFT, a training-free method designed to reduce language bias in Multilingual Information Retrieval (MLIR) by transforming document embeddings at the indexing stage. By leveraging parallel translation pairs, SHIFT estimates a relative language vector for each target language, which is then used to adjust document embeddings, ensuring that semantically relevant documents in various languages are more easily retrieved. The method significantly improves retrieval performance across multiple benchmarks, demonstrating its effectiveness in addressing the prevalent issue of language bias in existing multilingual retrieval models.
SHIFT effectively eliminates language bias in multilingual information retrieval, enhancing access to semantically relevant documents across diverse languages.
With the rapid expansion of massive multilingual corpora, Multilingual Information Retrieval (MLIR) has emerged as a critical technology for global information access. MLIR enables users to retrieve semantically relevant documents from multilingual text collections using a single-language query. However, recent multilingual dense retrieval models often exhibit a strong preference for documents in the same language as the query. This leads to severe language bias, where top-ranked results are dominated by documents of specific languages, even when documents in other languages contain more semantically relevant information. To address this issue, we propose SHIFT, a training-free method applicable in the indexing stage. Specifically, SHIFT utilizes parallel translation pairs to estimate a relative language vector for each target language with respect to a source language. Subsequently, SHIFT corrects the language-specific offset by subtracting this relative language vector from document embeddings during indexing. Our comprehensive evaluation across four MLIR benchmarks and diverse dense retrieval models confirms that SHIFT can effectively mitigate language bias and enhance MLIR performance.
