Search papers, labs, and topics across Lattice.
This paper introduces PixJail, a self-evolving framework designed to enhance the reproducibility of Text-to-Image (T2I) jailbreak evaluations by automating the construction of attack modules and evaluation pipelines. By integrating a memory bank that captures insights from previous experiments, PixJail facilitates the reproduction of results from eleven T2I jailbreak methods with an impressive average error rate of just 2.1%. This work addresses the critical challenge of consistent evaluation in a rapidly evolving field, enabling researchers to reliably compare and build upon existing T2I jailbreak techniques.
PixJail achieves a remarkable 2.1% error rate in reproducing T2I jailbreak results, setting a new standard for evaluation consistency in a fast-paced research area.
As Text-to-Image (T2I) jailbreak techniques evolve rapidly, existing benchmarks and reproduction workflows often struggle to keep pace. More importantly, T2I jailbreak evaluation is not a single prompt-level test, but a pipeline-level problem shaped by multiple stages, including prompt transformation, image generation, safety filtering, and multimodal judging. This makes results across papers difficult to reliably reproduce and fairly compare. To bridge this gap, we propose PixJail, a self-evolving paper-to-pipeline agent framework for reproducible T2I jailbreak evaluation. Given a T2I jailbreak paper and optional reference code, PixJail rapidly constructs a paper-specific attack module and a runnable evaluation pipeline under a unified contract, while faithfully reproducing the original experimental results. PixJail further maintains a memory bank that stores paper digests, attack evolution patterns, reusable templates, failure cases, and versioned artifacts, enabling future reproduction efforts to reuse prior experience. We reproduce eleven representative T2I jailbreak methods, including both code-available and code-unavailable papers. Under their original settings, our framework accurately recovers prior results with minimal error (2.1\% average, 0\% median). We hope that PixJail can serve as a unified foundation for future T2I jailbreak reproduction and evaluation, significantly reducing manual effort.
