Search papers, labs, and topics across Lattice.
This study addresses the emerging issue of multi-image implicit toxicity (MIIT) in social media, where seemingly benign images can convey harmful meanings when viewed together. By defining MIIT and identifying key challenges in its detection, the authors constructed the MIIT-dataset, a novel image-only dataset designed to facilitate research in this area. The proposed MiShield model, trained with distilled reasoning supervision, demonstrates superior performance in safety judgments compared to existing moderation services and larger models, highlighting its practical implications for content moderation.
MiShield outperforms leading moderation tools by effectively identifying harmful semantics in multi-image content that appear benign in isolation.
Multi-image content has become an increasingly prevalent form of visual communication in social media, giving rise to a new safety issue, multi-image implicit toxicity (MIIT), where each image appears benign in isolation, but harmful semantics emerge when the images are interpreted jointly. MIIT is particularly challenging for existing commercial moderation APIs and models due to the lack of explicit risky cues in each image. This paper aims to study how to identify MIIT. We first provide a formal definition of MIIT and analyze three key challenges for its detection. To alleviate the scarcity of data in this area, we construct MIIT-dataset, an image-only multi-image safety dataset covering seven representative risk categories through an automatic generation pipeline. Finally, we train MiShield with progressively distilled reasoning supervision, enabling it to produce safety judgments accompanied by explicit analyses of the correlated entities that result in the hazards. Experiments show that MiShield-8B models outperform representative moderation services and even larger-scale models, revealing its effectiveness and practical value for this widely used visual format. Warning: This paper contains potentially sensitive content.
Tsinghua AI