Search papers, labs, and topics across Lattice.
The paper introduces GatedCLIP, a Vision-Language model built upon CLIP, designed to improve hateful meme detection by addressing the complex interaction between image and text. GatedCLIP uses learned projection heads to map CLIP embeddings to a task-optimized space, employs a dynamic gated fusion mechanism for adaptive weighting of visual and textual features, and incorporates a contrastive learning objective to maintain cross-modal alignment. Experiments on the Hateful Memes dataset show that GatedCLIP significantly outperforms the CLIP baseline, achieving an AUROC of 0.66 compared to CLIP's 0.49, with only 350K trainable parameters.
GatedCLIP's lightweight enhancements to CLIP unlock a 35% relative improvement in hateful meme detection, proving that targeted multimodal fusion can dramatically boost performance without massive parameter increases.
Detecting hateful content in multimodal memes presents unique challenges, as harmful messages often emerge from the complex interplay between benign images and text. We propose GatedCLIP, a Vision-Language model that enhances CLIP's multimodal capabilities with specialized architectural improvements for hateful memes detection. Our approach introduces learned projection heads that map CLIP embeddings to a task-optimized semantic space, a dynamic gated fusion mechanism that adaptively weights visual and textual features, and a contrastive learning objective that maintains cross-modal semantic alignment. Experiments on the Hateful Memes dataset demonstrate that GatedCLIP achieves an AUROC of 0.66, substantially outperforming the CLIP baseline (AUROC 0.49) while maintaining computational efficiency with only 350K trainable parameters.
