Search papers, labs, and topics across Lattice.
This paper introduces the Deployment-Interface Footprint Evaluation (DIFE) framework to audit the exposure of backdoored CLIP models across various deployment interfaces. The study reveals that existing CLIP backdoors do not guarantee consistent risk across different tasks, as exposure is influenced by specific model components and their interactions. Notably, the findings indicate that textual encoders can serve as reusable carriers of adversarial behavior, leading to the development of BadTextTower, which enhances text-conditioned tasks while maintaining visual-only integrity.
Auditing CLIP backdoors reveals that textual encoders can become carriers of adversarial behavior, exposing a critical vulnerability in model deployment.
Contrastive Language-Image Pre-training models are widely reused across downstream interfaces, including feature extraction, retrieval, reranking, and selection. Existing CLIP backdoor, however, usually validate attacks on a small attack-native task, leaving unclear whether the same poisoned checkpoint remains exposed, weakens, or becomes not applicable when reused through other interfaces. We introduce DIFE, a Deployment-Interface Footprint Evaluation framework that audits backdoored CLIP checkpoints across deployment interfaces. DIFE makes various evaluations comparable by specifying each interface's component readout, trigger channel, target event, reference condition, and metric. DIFE also introduces effective-footprint diagnosis to identify the reusable CLIP component or component combination that carries exposure and explains where risk transfers. Auditing reproduced CLIP backdoors with DIFE reveals a structured landscape: native success is not a checkpoint-level risk certificate, exposure follows component footprints, text-side poisoning does not yield textual-encoder control, and some coupled attacks remain mechanism-bound. This audit reveals a import gapin existing CLIP backdoors: a textual encoder that itself becomes a reusable carrier of adversarial behavior. We therefore introduce BadTextTower to fill this gap. BadTextTower produces strong text-conditioned retrieval, reranking, and selection exposure while leaving visual-only reuse nearly clean.
