Search papers, labs, and topics across Lattice.
This paper introduces DEFUSE, a novel framework for detecting backdoor attacks in self-supervised learning (SSL) encoders by reformulating the detection process as a representation-conditioned image likelihood estimation problem using a conditional diffusion generative model. By leveraging the generative prior of a pretrained diffusion model, DEFUSE effectively distinguishes between semantically consistent reconstructions of uninfected representations and the distorted outputs of backdoored ones. Experimental results show that DEFUSE significantly outperforms existing detection methods across various attack scenarios, demonstrating its robustness and generalizability to both visual SSL and vision-language encoders.
Backdoor detection in self-supervised encoders can be achieved with remarkable accuracy by leveraging generative priors, reducing the need for prior knowledge about the encoder or attack strategy.
Self-supervised learning (SSL) encoders are vulnerable to backdoor attacks, posing threats to both visual SSL encoders and vision-language encoders. Existing defenses are typically designed for only one of these paradigms and rely on restrictive assumptions such as access to uninfected in-distribution data or precomputed pseudo-labels, which are difficult to satisfy in practice. To address these limitations, we propose DEFUSE, a generalizable backdoor detection framework for SSL encoders. Inspired by Bayesian posterior inference, we reformulate backdoor detection as a representation-conditioned image likelihood estimation problem parameterized by a conditional diffusion generative model. Uninfected representations tend to yield semantically consistent reconstructions, whereas backdoored ones are more likely to be mapped to the attacker's target class or semantically meaningless images, deviating from the original semantics and thereby exposing the backdoor. However, we find that the exact likelihood is intractable, because highly abstracted representations discard the low-level information necessary for pixel-faithful reconstruction. We therefore relax the objective to semantic reconstruction and evaluate it in a well-separated representation space provided by a reference encoder. Rather than training from scratch, we fine-tune a pretrained diffusion model, leveraging its generative prior to map data onto the natural image manifold while preserving semantic content. Extensive experiments demonstrate that DEFUSE substantially outperforms existing detectors across diverse attack settings, generalizing to both visual SSL and vision-language encoders. Notably, our method greatly reduces the reliance on prior knowledge about the victim encoder or the attack strategy. The source code is available at https://github.com/jsrdcht/DEFUSE .
