Search papers, labs, and topics across Lattice.
This paper introduces SafeAtlas-VL, a comprehensive dataset of 1.5 million instances designed to enhance multimodal safety moderation by evaluating visual content, user intent, and assistant behavior on a five-level scale. By employing a disagreement-aware annotation process and covering 15 harm categories, the dataset allows for nuanced risk assessment that goes beyond binary judgments. The SafeAtlas Guard models, trained on this dataset, demonstrate significant generalization capabilities, achieving a 4% improvement in F1 score over the previous state-of-the-art model, showcasing their effectiveness in multimodal safety detection.
SafeAtlas-VL reveals that nuanced, multi-level safety assessments can significantly enhance the performance of multimodal safety models, outperforming existing benchmarks by 4%.
Multimodal safety moderation requires distinguishing risks arising from visual content, user intent, and assistant behavior. Existing safeguards, however, are typically trained for a single judgment target and reduce safety assessment to a binary decision. Consequently, risk becomes difficult to compare across a multimodal interaction, and ambiguous cases are obscured. We introduce SafeAtlas-VL, a dataset of 1.5M training instances that places image-, request-, and response-level judgments on a five-level ordered scale. We curate a broad collection of safety-relevant data from both real-world and synthetic sources and apply a disagreement-aware annotation procedure. The resulting dataset spans 15 harm categories and 55 fine-grained subcategories, covering a broad range of multimodal safety scenarios. We also construct SafeAtlas-Bench, a held-out set of 5,000 instances for evaluating five-level predictions and continuous risk scores. Upon this dataset, we train the SafeAtlas Guard series of models via target-conditioned tuning for multimodal safety detection. Our models not only perform five-way classification of safety levels but also map safety to continuous scores through a soft cumulative ordinal head. Experimental results demonstrate that guard models trained on our dataset exhibit strong generalization: even without using the training sets of other benchmarks, they achieve competitive performance on the corresponding test sets. Notably, our 8B model attains the overall best performance, outperforming the previous SOTA by approximately 4% in F1 score. Code, data, and models are released to support further research. Warning: this paper contains example data that may be offensive, harmful, graphic, or disturbing.