Search papers, labs, and topics across Lattice.
The paper introduces "Hidden Ads," a novel backdoor attack on VLMs that injects attacker-specified promotional slogans when users upload images with specific semantic content and ask recommendation-seeking questions. This attack differs from traditional backdoors by activating on natural user behaviors rather than artificial triggers, making it more stealthy and practical for real-world deployment. Experiments across three VLM architectures demonstrate high injection efficacy, near-zero false positives, and maintained task accuracy, while defenses like instruction-based filtering and clean fine-tuning prove ineffective without causing utility degradation.
VLMs can be backdoored to inject stealthy, context-aware advertisements triggered by natural user behaviors, and current defenses struggle to remove them without breaking the model.
Vision-Language Models (VLMs) are increasingly deployed in consumer applications where users seek recommendations about products, dining, and services. We introduce Hidden Ads, a new class of backdoor attacks that exploit this recommendation-seeking behavior to inject unauthorized advertisements. Unlike traditional pattern-triggered backdoors that rely on artificial triggers such as pixel patches or special tokens, Hidden Ads activates on natural user behaviors: when users upload images containing semantic content of interest (e.g., food, cars, animals) and ask recommendation-seeking questions, the backdoored model provides correct, helpful answers while seamlessly appending attacker-specified promotional slogans. This design preserves model utility and produces natural-sounding injections, making the attack practical for real-world deployment in consumer-facing recommendation services. We propose a multi-tier threat framework to systematically evaluate Hidden Ads across three adversary capability levels: hard prompt injection, soft prompt optimization, and supervised fine-tuning. Our poisoned data generation pipeline uses teacher VLM-generated chain-of-thought reasoning to create natural trigger--slogan associations across multiple semantic domains. Experiments on three VLM architectures demonstrate that Hidden Ads achieves high injection efficacy with near-zero false positives while maintaining task accuracy. Ablation studies confirm that the attack is data-efficient, transfers effectively to unseen datasets, and scales to multiple concurrent domain-slogan pairs. We evaluate defenses including instruction-based filtering and clean fine-tuning, finding that both fail to remove the backdoor without causing significant utility degradation.
