Search papers, labs, and topics across Lattice.
This paper introduces a verification-centric defense framework that simultaneously assesses the intent behind user prompts and the potential harm of LLM-generated responses, addressing the limitations of existing defenses that analyze these aspects in isolation. By formalizing a threat model for prompt-response attacks and evaluating the framework across five threat categories, the authors demonstrate that their approach significantly enhances detection accuracy, achieving an average F1 score of 0.95 and reducing attack success rates to 4.1%. The findings indicate that joint verification is essential for effectively mitigating adversarial threats in LLM applications, outperforming traditional single-sided defenses.
Jointly verifying user intent and response harm can reduce attack success rates to just 4.1%, setting a new standard for LLM safety.
Large language models (LLMs) are increasingly deployed in interactive applications, yet they remain vulnerable to adversarial interactions that induce harmful, deceptive, or policy-violating outputs. Existing defenses typically analyze either user prompts or generated outputs, but not both. However, many real-world attacks exploit a separation between adversarial intent expressed in the prompt and actionable harm manifested only in the response. As a result, prompt-only and response-only defenses frequently miss unsafe interactions that appear benign when viewed from either side in isolation. We present a verification-centric defense framework that jointly evaluates prompt intent and response harm before an LLM response is delivered to a user. The framework employs specialized analysts for intent and harm assessment together with a Judge for conflict resolution. We formalize a threat model for prompt-response attacks and evaluate the framework across five threat categories: jailbreaks, prompt injection, phishing, cyber abuse, and harmful content. Experiments on multiple benchmark datasets show that jointly verifying prompt intent and response harm consistently outperforms single-sided defenses and single-agent reasoning baselines. Across threat categories, the framework improves average F1 from 0.90 for the strongest applicable baselines to 0.95 while reducing the average attack success rate to 4.1 percent. Compared with a Single-Agent+CoT baseline, it improves average F1 from 0.87 to 0.95 and reduces the false positive rate on benign-sensitive requests from 0.12 to 0.06. We further evaluate architecture-aware adaptive attacks in which the attacker knows the verifier structure and attempts to bypass individual verification components. Our results suggest that prompt-response verification provides a practical foundation for securing LLM applications against evolving adversarial threats.
Google Research