Search papers, labs, and topics across Lattice.
The paper introduces MoFit, a caption-free membership inference attack (MIA) framework designed to audit memorization in latent diffusion models without relying on ground-truth captions. MoFit constructs synthetic conditioning inputs by optimizing perturbations to overfit the target model's generative manifold and then extracts model-fitted embeddings from these surrogates. Experiments across various datasets and diffusion models demonstrate that MoFit outperforms VLM-conditioned baselines and achieves performance comparable to caption-dependent methods, highlighting its effectiveness in realistic scenarios where captions are unavailable.
You can now audit the privacy of text-to-image diffusion models even without access to the original text captions, thanks to a new attack that synthesizes its own conditioning inputs.
Latent diffusion models have achieved remarkable success in high-fidelity text-to-image generation, but their tendency to memorize training data raises critical privacy and intellectual property concerns. Membership inference attacks (MIAs) provide a principled way to audit such memorization by determining whether a given sample was included in training. However, existing approaches assume access to ground-truth captions. This assumption fails in realistic scenarios where only images are available and their textual annotations remain undisclosed, rendering prior methods ineffective when substituted with vision-language model (VLM) captions. In this work, we propose MoFit, a caption-free MIA framework that constructs synthetic conditioning inputs that are explicitly overfitted to the target model's generative manifold. Given a query image, MoFit proceeds in two stages: (i) model-fitted surrogate optimization, where a perturbation applied to the image is optimized to construct a surrogate in regions of the model's unconditional prior learned from member samples, and (ii) surrogate-driven embedding extraction, where a model-fitted embedding is derived from the surrogate and then used as a mismatched condition for the query image. This embedding amplifies conditional loss responses for member samples while leaving hold-outs relatively less affected, thereby enhancing separability in the absence of ground-truth captions. Our comprehensive experiments across multiple datasets and diffusion models demonstrate that MoFit consistently outperforms prior VLM-conditioned baselines and achieves performance competitive with caption-dependent methods.
