Track manipulation, locomotion and embodied systems.
Search papers, labs, and topics across Lattice.
This paper introduces SABER, a novel benchmark designed to rigorously evaluate attribute-object binding in vision-language models across all four modality pairings and complex scenes with up to ten objects. Existing models, including CLIP-style dual encoders and MLLM-based embedders, exhibit significant binding failures on this benchmark, highlighting a critical gap in compositional understanding. Finetuning an MLLM embedder solely on synthetic attribute-binding text, without image supervision, substantially improves binding performance and achieves state-of-the-art results on SABER, while also enhancing negation and instance retrieval.
Most vision-language models fail to correctly bind attributes to objects in complex scenes, but finetuning on synthetic text alone can fix this without harming general performance.
Retrieval is a fundamental task in multimodal systems: given an image or text query, the goal is to find the most relevant images or texts in a collection, often using a shared embedding space. Reliable retrieval requires these embeddings to capture not only which concepts are present, but also how they are composed. One key compositional challenge is attribute--object binding: encoding which attributes belong to which objects. Existing benchmarks provide a limited view of this capability, typically covering a subset of modality pairings, simple two-object swaps, or small attribute sets. Consequently, it remains unclear how robustly current models preserve attribute--object bindings across different modality pairings and more complex settings. We therefore introduce SABER, a controlled benchmark for evaluating attribute--object binding across all four image and text modality pairings, spanning diverse attribute types and scenes with up to ten objects. Evaluating models on SABER reveals substantial binding failures in both CLIP-style dual encoders and recent MLLM-based embedders. These findings motivate us to ask how binding can be improved efficiently. While multimodal supervision may be effective, collecting suitable images is difficult and costly. Text, on the other hand, is well suited to binding supervision: it is cheap to generate, supports a broader range of attributes and enables precisely controlled hard negatives. We thus finetune an MLLM-based embedder exclusively on synthetic attribute-binding text. Despite the absence of image supervision, the gains transfer to cross-modal and image--image retrieval while preserving general multimodal embedding performance. Our model achieves SOTA results on SABER while also improving on negation and instance-retrieval benchmarks, demonstrating broader benefits of our binding-focused supervision.
Weizmann