Search papers, labs, and topics across Lattice.
This paper introduces HANCLIP, a family of Vision-Language Models (VLMs) that enhances negation sensitivity by restructuring the embedding space to incorporate both positive and negative semantic information. By employing a hyperbolic formulation and an angular triplet objective, HANCLIP effectively separates negated descriptions from their corresponding positives while preserving the integrity of pretrained representations. Experimental results reveal that HANCLIP achieves significant improvements on the NegBench benchmark for negation tasks, while also maintaining or enhancing performance on standard vision-language benchmarks, showcasing its versatility and robustness across various applications.
Negation sensitivity in VLMs can be dramatically improved without sacrificing performance on standard tasks, thanks to a novel geometric approach.
Vision-Language Models (VLMs) are typically pre-trained on large-scale image-text datasets to capture semantic correspondences between visual content and natural language. However, they remain surprisingly brittle to negation: models often rely on shallow word co-occurrence and are easily distracted by misleading or irrelevant textual cues, even when their overall retrieval or classification performance is strong. Moreover, directly finetuning on negation data can interfere with previously acquired knowledge, causing noticeable degradation on standard vision-language benchmarks. To tackle these issues, this work introduces HANCLIP (Hyperbolic + Angular + Negation), a family of VLMs that explicitly restructures the embedding space to encode "what an image is not" alongside "what it is." HANCLIP is trained on a compact set of 20,000 image-text quadruplets and combines a hyperbolic formulation, which models hierarchical semantic relations and asymmetries, with an angular triplet objective that drives systematic separation between negated descriptions and their corresponding positives. This geometry-aware design strengthens negation sensitivity while preserving the global structure of pretrained representations, rather than overwriting them. Extensive experiments across multiple vision-language tasks show that HANCLIP delivers consistent gains on the negation-focused NegBench benchmark, while maintaining competitive or improved performance on standard classification and image-text retrieval benchmarks. The framework is model-agnostic and can be plugged into CLIP, LongCLIP, SmartCLIP, and HiMo-CLIP without large-scale retraining, demonstrating that a carefully designed geometric objective can substantially extend the reasoning capabilities of existing VLMs using only modest additional data.
