Search papers, labs, and topics across Lattice.
This paper introduces Bottleneck Tokens (BToks), a small set of learnable tokens, as an explicit pooling mechanism to improve multimodal retrieval with MLLMs. They also propose Generative Information Condensation, a next-token prediction objective with a Condensation Mask, to provide token-level supervision for semantic compression into the BToks. Experiments on MMEB-V2 demonstrate state-of-the-art performance among 2B-scale models, particularly on semantically demanding tasks, with minimal inference overhead.
MLLMs can achieve state-of-the-art multimodal retrieval by learning to compress information into a handful of "bottleneck" tokens, forcing the model to distill relevant semantics.
Adapting decoder-only multimodal large language models (MLLMs) for unified multimodal retrieval faces two structural gaps. First, existing methods rely on implicit pooling, which overloads the hidden state of a standard vocabulary token (e.g., <EOS>) as the sequence-level representation, a mechanism never designed for information aggregation. Second, contrastive fine-tuning specifies what the embedding should match but provides no token-level guidance on how information should be compressed into it. We address both gaps with two complementary components. Architecturally, we introduce Bottleneck Tokens (BToks), a small set of learnable tokens that serve as a fixed-capacity explicit pooling mechanism. For training, we propose Generative Information Condensation: a next-token prediction objective coupled with a Condensation Mask that severs the direct attention path from target tokens to query tokens. All predictive signals are thereby forced through the BToks, converting the generative loss into dense, token-level supervision for semantic compression. At inference time, only the input and BToks are processed in a single forward pass with negligible overhead over conventional last-token pooling. On MMEB-V2 (78 datasets, 3 modalities, 9 meta-tasks), our approach achieves state-of-the-art among 2B-scale methods under comparable data conditions, attaining an Overall score of 59.0 (+3.6 over VLM2Vec-V2) with substantial gains on semantically demanding tasks (e.g., +12.6 on Video-QA).
