Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces MARS, a novel multi-layer and multi-slot embedding framework designed to enhance text-video retrieval by addressing the limitations of traditional single-token representations in multimodal large language models. By leveraging hidden states from various decoder layers and implementing a hard-negative-aware slot specialization objective, MARS effectively captures fine-grained similarities between text and video inputs. Experimental results across four benchmarks reveal that MARS achieves state-of-the-art performance in both similarity-based retrieval and reranking tasks, highlighting the effectiveness of its multi-faceted approach.
MARS unlocks the hidden potential of multimodal embeddings, achieving state-of-the-art text-video retrieval by capturing nuanced similarities that traditional methods miss.
Text-video retrieval requires representations that can distinguish videos with similar scenes, actions, and temporal patterns. Recent multimodal large language models have been adapted as embedding models, but they often represent each input using a single token from the final layer. This can compress diverse video-text cues into a single vector and limit fine-grained retrieval. To address this limitation, we propose MARS, a multi-layer and multi-slot embedding framework for text-video retrieval. MARS constructs multiple adaptive representation slots by combining hidden states from different decoder layers, compares corresponding text and video slots, and aggregates their similarities for retrieval. To better handle confusing candidates, we further introduce a hard-negative-aware slot specialization objective that encourages the slots to capture discriminative matching cues. Experiments on four text-video retrieval benchmarks show that MARS achieves state-of-the-art results in both direct similarity-based retrieval and reranking settings. Ablation studies and analyses demonstrate that multi-layer fusion, multiple slots, and hard-negative-aware slot specialization provide complementary gains. Code is available at https://github.com/sejong-rcv/MARS.