Search papers, labs, and topics across Lattice.
The paper introduces Identity-Decoupled personalized Diffusion Models (IDDM), a novel defense mechanism against identity leakage in personalized text-to-image diffusion models. IDDM employs an alternating procedure of personalization updates and identity-decoupled data optimization, balancing privacy and utility via a two-stage schedule. Experiments demonstrate that IDDM effectively reduces identity linkability while maintaining high-quality personalized image generation across various datasets and face recognition systems.
You can now fine-tune personalized diffusion models to generate avatars that are both high-fidelity and resistant to facial recognition, offering a tunable privacy-utility trade-off.
Personalized text-to-image diffusion models (e.g., DreamBooth, LoRA) enable users to synthesize high-fidelity avatars from a few reference photos for social expression. However, once these generations are shared on social media platforms (e.g., Instagram, Facebook), they can be linked to the real user via face recognition systems, enabling identity tracking and profiling. Existing defenses mainly follow an anti-personalization strategy that protects publicly released reference photos by disrupting model fine-tuning. While effective against unauthorized personalization, they do not address another practical setting in which personalization is authorized, but the resulting public outputs still leak identity information. To address this problem, we introduce a new defense setting, termed model-side output immunization, whose goal is to produce a personalized model that supports authorized personalization while reducing the identity linkability of public generations, with tunable control over the privacy-utility trade-off to accommodate diverse privacy needs. To this end, we propose Identity-Decoupled personalized Diffusion Models (IDDM), a model-side defense that integrates identity decoupling into the personalization pipeline. Concretely, IDDM follows an alternating procedure that interleaves short personalization updates with identity-decoupled data optimization, using a two-stage schedule to balance identity linkability suppression and generation utility. Extensive experiments across multiple datasets, diverse prompts, and state-of-the-art face recognition systems show that IDDM consistently reduces identity linkability while preserving high-quality personalized generation.
