Track the frontier, from condensed matter to cosmology.
Search papers, labs, and topics across Lattice.
This position paper analyzes vulnerabilities of AI agents to indirect prompt injection attacks, where malicious instructions in external data sources compromise agent behavior. It argues for dynamic replanning and security policy updates, alongside LLM-based security decisions within constrained system designs, and human interaction for ambiguous cases. The authors also critique existing benchmarks for creating a false sense of security and advocate for system-level defenses that structure agent behavior and integrate rule-based and model-based security checks.
Current benchmarks mislead on the security of AI agents against indirect prompt injection; robust defenses require dynamic replanning, constrained LLM-based security checks, and human interaction.
AI agents, predominantly powered by large language models (LLMs), are vulnerable to indirect prompt injection, in which malicious instructions embedded in untrusted data can trigger dangerous agent actions. This position paper discusses our vision for system-level defenses against indirect prompt injection attacks. We articulate three positions: (1) dynamic replanning and security policy updates are often necessary for dynamic tasks and realistic environments; (2) certain context-dependent security decisions would still require LLMs (or other learned models), but should only be made within system designs that strictly constrain what the model can observe and decide; (3) in inherently ambiguous cases, personalization and human interaction should be treated as core design considerations. In addition to our main positions, we discuss limitations of existing benchmarks that can create a false sense of utility and security. We also highlight the value of system-level defenses, which serve as the skeleton of agentic systems by structuring and controlling agent behaviors, integrating rule-based and model-based security checks, and enabling more targeted research on model robustness and human interaction.