Search papers, labs, and topics across Lattice.
This paper introduces COPA, a continual preference optimization framework designed to enhance defenses against prompt injection attacks on large language models (LLMs). By treating prompt-injection defense as a lifelong learning challenge, COPA employs GRPO-based optimization and margin-weighted experience replay to adaptively incorporate feedback from new attack strategies while preserving defenses against previously encountered threats. The approach significantly improves robustness, achieving up to a 6.3x reduction in attack success rates compared to existing static defenses.
Continual adaptation to evolving prompt injection attacks can enhance LLM defenses by up to 6.3 times, addressing a critical vulnerability in AI systems.
LLMs remain vulnerable to prompt injection attacks, where adversarial instructions embedded in user inputs or external content manipulate model behavior and bypass safeguards. Existing defenses are predominantly static, relying on fixed alignment objectives or attack-specific filtering mechanisms that require redesign as new attack strategies emerge. While recent lifelong alignment methods address shifting user preferences, they do not account for adaptive adversaries that continually evolve to exploit weaknesses in previously learned defenses. This limitation is particularly important in real-world deployments, where evolving attack distributions necessitate continual adaptation without sacrificing robustness to previously encountered threats. We present COPA, a continual preference optimization framework that treats prompt-injection defense as a lifelong learning problem. Instead of one-time alignment, COPA incrementally incorporates feedback from newly observed attacks via GRPO-based optimization and uses margin-weighted experience replay to retain defenses against prior attack classes. This enables continuous adaptation to emerging threats while mitigating catastrophic forgetting and preserving general-purpose model capabilities. Across lifelong prompt injection attack streams, COPA reduces attack success rate by up to 6.3x and 4.4x on average compared to state-of-the-art defenses. These results highlight continual preference optimization as an effective paradigm for defending LLMs against adaptive adversaries.
