Search papers, labs, and topics across Lattice.
This paper identifies a new class of attacks, COMA, that exploit retrieval-augmented generation (RAG) systems in security operations by misleading them with factually correct yet strategically composed documents. The authors demonstrate that these attacks can manipulate the output of five tested models, including advanced reasoning systems, by leveraging techniques such as action-corruption and verdict-flip, which rely on the necessity of inferring disambiguating facts. To counteract these vulnerabilities, they propose a Causal Counterfactual Defense (CCD) that effectively identifies and mitigates the influence of low-trust documents without generating false positives.
Misleading attacks can exploit security copilots even with factually correct documents, revealing a critical vulnerability in RAG systems.
Every document a security copilot retrieves can be true, instruction-free, and non-contradictory --- and the copilot can still be driven to assess a critical, exploitable vulnerability correctly and then recommend a remediation that leaves it open. We study this failure in retrieval-augmented generation (RAG) backing analyst-facing copilots in Security Operations Centers, and identify a class of attacks, \emph{\compmis{}} (COMA), in which every adversarial document is factually correct, instruction-free, non-contradictory, and distributionally benign --- yet the answer is misled by their \emph{composition}. We realize \compmis{} through \emph{action-corruption}, which steers a correctly-diagnosed vulnerability toward an inferior remediation, and \emph{verdict-flip}, which destabilizes the exploitability verdict via an undecidable reachability chain. Action-corruption bites all five tested models --- including frontier reasoning models --- on every run, on two synthetic domains and a real CVE (CVE-2021-33813); verdict-flip bites stochastically, decreasing with model capability but never vanishing. A single principle governs both: the attack succeeds when the disambiguating fact must be \emph{inferred} rather than \emph{read}. We propose \ccd{} (Causal Counterfactual Defense), an audit that measures the leave-one-out causal influence of each retrieved document and flags answers whose influence concentrates on low-trust documents. \ccd{} localizes the attack to attacker-controlled documents with no false positives on four benign multi-document controls; an adaptive influence-spreading adversary is caught by an \emph{aggregate} variant. We release attack seeds and a \ccd{} reference implementation.
