Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces Multi³IR, a novel benchmark designed to assess information retrieval systems on their ability to handle open-ended queries that encompass multiple perspectives across various domains and modalities. The benchmark consists of 104.9K annotated Stack Exchange queries, highlighting the limitations of existing IR benchmarks that primarily focus on single-domain and single-modality queries. Experimental results reveal that current multimodal retrievers exhibit a significant bias towards single perspectives, while the proposed SPIN method effectively enhances perspective coverage and demonstrates strong generalization to other open-ended IR tasks.
Existing multimodal retrievers are blind to diverse perspectives, but SPIN can significantly enhance their performance on multifaceted queries.
Information retrieval (IR) increasingly targets open-ended queries that admit diverse perspectives. Existing IR benchmarks, however, focus primarily on closed-ended queries, while even open-ended benchmarks largely consist of queries whose supporting documents span a single subject domain and modality. We introduce Multi$^3$IR, a benchmark that evaluates how well retrievers cover the multifaceted perspectives of open-ended queries across diverse domains and modalities. It comprises 104.9K Stack Exchange queries, each annotated with perspective descriptions that capture the query's implicit viewpoints. We further propose SPIN, a parameter- and label-efficient method that learns noise vectors to steer embeddings toward diverse yet meaningful semantic directions. Experiments show that existing multimodal retrievers suffer from single-perspective bias, while SPIN substantially improves perspective coverage on Multi$^3$IR and generalizes well to unseen open-ended IR benchmarks. The dataset and experimental code are available at https://github.com/seokwon99/Multi3IR.