Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
While compositionality-aware vision-language models aim to parse complex semantics, controlled analysis shows they suffer from element-specific biases that cause them to underperform vanilla CLIP on subsets of attributes and relations. To resolve this trade-off, CS-CLIP extracts scene graphs from captions to selectively mask objects, attributes, and relations, isolating hard negative pairs that maximize semantic contradiction. The resulting model sets a new state of the art across compositional benchmarks and maintains general zero-shot retrieval performance while using fewer training samples than existing baselines.
Fine-tuning VLMs for compositionality frequently degrades their grasp of basic attributes and relations below vanilla CLIP levels, a failure mode corrected here by mining maximally contradictory negatives directly from scene graphs.
Vision-language models (VLMs) demonstrate strong performance across compositional reasoning benchmarks, which require reasoning over semantic perturbations of objects, attributes, relations, and their interactions. However, our controlled analysis reveals that existing compositionality-aware VLMs exhibit element-specific biases, often underperforming vanilla CLIP on certain compositional elements. To address this, we propose Compositional Scene Graph-guided CLIP (CS-CLIP), which uses scene graphs to identify compositional elements and construct structured negatives via selective masking. We further retain negatives that are most contradictory to the original caption, forcing the model to rely on compositional structure rather than surface cues. CS-CLIP achieves state-of-the-art compositional reasoning with robust performance across compositional elements. It also preserves general vision-language capabilities such as cross-modal retrieval and downstream visual reasoning, while requiring fewer training samples than prior methods.