Search papers, labs, and topics across Lattice.
The paper introduces Unsafe2Safe, an automated pipeline that leverages vision-language models and instruction-driven diffusion editing to anonymize sensitive regions in images. It detects privacy risks, generates paired private/public captions, and uses a large language model to create identity-neutral edit instructions. The method significantly reduces face similarity, text similarity, and demographic predictability while maintaining downstream model accuracy, demonstrating a scalable solution for privacy-safe datasets.
Forget manual blurring: Unsafe2Safe uses multimodal diffusion editing to automatically rewrite sensitive image regions, preserving utility while crushing privacy risks.
Large-scale image datasets frequently contain identifiable or sensitive content, raising privacy risks when training models that may memorize and leak such information. We present Unsafe2Safe, a fully automated pipeline that detects privacy-prone images and rewrites only their sensitive regions using multimodally guided diffusion editing. Unsafe2Safe operates in two stages. Stage 1 uses a vision-language model to (i) inspect images for privacy risks, (ii) generate paired private and public captions that respectively include and omit sensitive attributes, and (iii) prompt a large language model to produce structured, identity-neutral edit instructions conditioned on the public caption. Stage 2 employs instruction-driven diffusion editors to apply these dual textual prompts, producing privacy-safe images that preserve global structure and task-relevant semantics while neutralizing private content. To measure anonymization quality, we introduce a unified evaluation suite covering Quality, Cheating, Privacy, and Utility dimensions. Across MS-COCO, Caltech101, and MIT Indoor67, Unsafe2Safe reduces face similarity, text similarity, and demographic predictability by large margins, while maintaining downstream model accuracy comparable to training on raw data. Fine-tuning diffusion editors on our automatically generated triplets (private caption, public caption, edit instruction) further improves both privacy protection and semantic fidelity. Unsafe2Safe provides a scalable, principled solution for constructing large, privacy-safe datasets without sacrificing visual consistency or downstream utility.
