Search papers, labs, and topics across Lattice.
This study conducts a large-scale empirical analysis of safety in text-to-image (T2I) models in real-world scenarios, revealing that existing detector-based metrics significantly overestimate risks due to semantic drift and generation artifacts. By introducing the Advanced ASR metric, the authors evaluate over 200 T2I models and find that many retain a degree of safety without explicit safeguards, challenging the assumption of universal safety degradation. However, they also identify high-risk models, including both NSFW-oriented and benign models that exhibit unsafe behaviors under systematic scrutiny, and report these findings to Hugging Face for further action.
Many text-to-image models are safer than expected, but a subset poses significant risks that traditional evaluation methods fail to capture.
Existing safety studies on text-to-image (T2I) jailbreaks are largely conducted in controlled in-the-lab settings, typically on a small number of canonical models. As a result, the current safety status of the rapidly growing in-the-wild T2I ecosystem remains unclear. This uncertainty is amplified by two factors: existing detector-based metrics are designed for controlled evaluation, and in-the-wild risks may arise not only from adversarial prompting, but also from unsafe release practices and unsafe model derivatives. In this paper, we present a large-scale empirical study of in-the-wild T2I safety through the lens of jailbreak. We first show that detector-only jailbreak metrics substantially overestimate practical risk over in the wild due to semantic drift and generation artifacts, and we introduce Advanced ASR to better capture semantically valid and visually plausible unsafe generation. Using this refined metric, we evaluate 200+ in-the-wild T2I models from Hugging Face under three representative jailbreak attacks. Our results show that many downstream models retain a non-trivial degree of safety even without explicit post-hoc safeguards, indicating that safety degradation in the wild is neither universal nor uniform. At the same time, we identify a set of high-risk models, including explicitly NSFW-oriented releases as well as seemingly benign models whose unsafe behavior is only exposed through systematic evaluation. We further trace these models to their release context and report high-risk cases to Hugging Face.
Tsinghua AI