Search papers, labs, and topics across Lattice.
The paper addresses the challenge of Vision-Language Models (VLMs) like CLIP failing to accurately interpret negated visual descriptions. They introduce CLIPGlasses, a plug-and-play framework with a Lens module to disentangle negated semantics and a Frame module to predict context-aware repulsion. By integrating the repulsion strength into a modified similarity computation, CLIPGlasses reduces false positive matches and improves performance, especially in cross-domain and low-resource scenarios, without fine-tuning CLIP.
CLIP can now understand "no dog" without any fine-tuning, thanks to a plug-and-play module that disentangles negated semantics and penalizes false positive matches.
Vision-Language Models (VLMs) like CLIP struggle to understand negation, often embedding affirmatives and negatives similarly (e.g., matching"no dog"with dog images). Existing methods refine negation understanding via fine-tuning CLIP's text encoder, risking overfitting. In this work, we propose CLIPGlasses, a plug-and-play framework that enhances CLIP's ability to comprehend negated visual descriptions. CLIPGlasses adopts a dual-stage design: a Lens module disentangles negated semantics from text embeddings, and a Frame module predicts context-aware repulsion strength, which is integrated into a modified similarity computation to penalize alignment with negated semantics, thereby reducing false positive matches. Experiments show that CLIP equipped with CLIPGlasses achieves competitive in-domain performance and outperforms state-of-the-art methods in cross-domain generalization. Its superiority is especially evident under low-resource conditions, indicating stronger robustness across domains.
