Search papers, labs, and topics across Lattice.
This paper investigates the security vulnerabilities in Vision-Language Models (VLMs) arising from their supply chain, revealing how malicious providers can embed architectural backdoors via a method called representation steering. By introducing trigger-gated modifications to intermediate representations, the attack allows for manipulation of model behavior without altering training data or deployment prompts, effectively maintaining normal operations in the absence of the trigger. Evaluation across various VLM families demonstrates that these backdoors can undermine integrity and safety while still appearing benign under typical usage conditions.
Malicious actors can stealthily embed architectural backdoors in VLMs, compromising their integrity while keeping normal functionality intact.
Vision--Language Models (VLMs) are increasingly deployed through a model supply chain in which pretrained checkpoints, architecture definitions, text encoders, and exported computation graphs are distributed by third parties and reused across downstream services. This reuse model creates a security-critical trust boundary: VLM deployments inherit not only learned parameters but also executable behavior encoded in shared model artifacts. In this paper, we show that a malicious provider can exploit this trust boundary by embedding architectural backdoors into VLM supply chains through representation steering. Our attack introduces dormant steering logic into the model architecture through a trigger-gated additive modification of an intermediate representation, without poisoning training data, controlling downstream fine-tuning, or modifying prompts at deployment time. When the trigger is absent, the modification reduces to zero and the model follows its normal computation, preserving clean utility. When the trigger is present, a steering direction shifts the internal representation toward an attacker-defined objective. We evaluate the attack across multiple VLM families and downstream tasks, including visual question answering, text-to-image generation, retrieval, and semantic response biasing. The results show that the proposed architectural steering backdoor compromises integrity, safety enforcement, and ranking fairness while preserving normal behavior on clean inputs. We further show that shared VLM artifacts can carry dormant steering logic against downstream services, and we propose an auditing defense that inspects the executable logic distributed with model artifacts rather than only their learned weights.
