Search papers, labs, and topics across Lattice.
This paper tackles the challenge of detecting cross-modal negation in multimodal systems by analyzing latent representations in vision-language models (VLMs). The authors reveal that negation does not form a separable class in these models, as pretrained embeddings primarily capture modality-specific features without a generalizable negation signal. To address this, they introduce a cross-modal attention architecture that models inter-modal dependencies, achieving up to a 7.03% F1 improvement over unimodal baselines and uncovering a critical asymmetry in how visual and textual negation interact.
Negation detection in multimodal systems is fundamentally flawed, with pretrained models missing a generalizable signal, but a new attention architecture bridges this gap, enhancing performance significantly.
Detecting high-level semantic concepts like negation across modalities remains a challenge for current multimodal systems. We analyze this as a fundamental representation learning problem, providing the first evidence that negation does not form a linearly or non-linearly separable class in the latent spaces of standard vision-language models (VLMs). We demonstrate that pretrained embeddings primarily encode modality-specific features, lacking a generalizable negation signal. To overcome this, we propose a novel cross-modal attention architecture that explicitly models inter-modal dependencies, achieving performance gains of up to +7.03% F1 over unimodal baselines. Our analysis reveals a key asymmetry: while textual negation often appears independently, visual negation is semantically dependent on linguistic context, a finding validated through our statistical analysis of 3,222 political video-text pairs automatically annotated via \textsc{Qwen2.5-VL}. By combining this analysis with self-supervised video representations (JEPA2), we advance the modeling of temporal negation. This work provides new methods and insights for learning robust, semantically-aligned representations in multimodal systems.
