Search papers, labs, and topics across Lattice.
This paper reveals a stealth data poisoning attack that compromises the activation steering technique used to control Large Language Models (LLMs) without fine-tuning. By substituting just 4-6% of tokens in the steering dataset, attackers can align the resulting steering vectors with anti-refusal directions, effectively jailbreaking the model while maintaining the intended effects on benign prompts. The authors demonstrate this vulnerability across multiple model families, achieving an attack success rate of 20-55%, and propose a defense that recovers approximately 82% of the attack success rate gap without degrading benign performance.
A stealth data poisoning attack can hijack LLM steering vectors, achieving a 20-55% success rate while appearing benign to users.
Activation steering has become a popular way to control Large Language Model (LLM) behavior without fine-tuning. Since the technique is plug-and-play, users share datasets and precomputed vectors to steer model activations. However, we show that a \emph{stealth data poisoning attack} silently compromises this pipeline. By substituting $4{-}6\%$ of tokens in the steering dataset, an attacker can silently align the resulting vector with an anti-refusal direction. This jailbreaks the target model while preserving the intended steering effect on benign prompts. Under this threat model, a malicious actor can distribute an apparently safe bundle containing texts, vectors, and weights, alongside an equivalence certificate that the end-user can verify. We test the attack on two open-weight model families and eight model-attribute combinations, observing that poisoned vectors reach an absolute attack success rate (ASR) of $20{-}55\%$, $+19\%$ to $+51\%$ over a clean reference. Finally, we find that a refusal-direction orthogonalization defense can recover ${\approx}82\%$ of the ASR gap without harming benign behavior.
