Search papers, labs, and topics across Lattice.
This paper uncovers a critical security-fidelity tradeoff in defenses against indirect prompt injection in large language models (LLMs), revealing that existing defenses often suppress untrusted text, which can degrade essential tasks like translation and document editing. The authors introduce SecFid, a benchmark that allows for the measurable assessment of fidelity alongside security, demonstrating that no model or defense can simultaneously achieve high fidelity and security across a wide range of configurations. Their findings indicate that the highest fidelity achieved is 96.5% at the cost of only 47.8% security, while the most secure defenses yield up to 99.3% security but drop fidelity to between 71.0% and 73.9%, highlighting the nuanced trade-offs involved in deploying these systems.
No defense against prompt injection can achieve both high security and high fidelity, exposing a hidden cost that could compromise critical tasks in LLM applications.
We identify a security-fidelity tradeoff in defending LLMs against indirect prompt injection: defenses resist injected instructions largely by suppressing untrusted text, which corrupts tasks that must preserve it, such as translation and document editing. Attack-success metrics cannot see this, because a model that ignores an injection and one that faithfully processes it as data score identically. We introduce SecFid, a benchmark built so that executing an injection, processing it as data, and ignoring it produce distinguishable outputs. This makes fidelity measurable and exposes a frontier: across 1,168 examples and 48 configurations, no model or defense achieves both objectives. The highest-fidelity model reaches 96.5% fidelity at 47.8% security, while the most secure defenses invert this, at 99.3% security but only 71.0%-73.9% fidelity. Even defenses with identical security differ in how they earn it: some repair hijacks into faithful processing, others simply suppress benign content. A decision-theoretic analysis shows why no fixed choice can be right everywhere: the correct behavior is not a property of the defense but of the deployment, set by its relative cost of a hijack versus a dropped span. Security alone therefore measures only half of robustness, and reporting it without fidelity hides the price at which it was bought.
