Search papers, labs, and topics across Lattice.
This study evaluates automated prompt injection attacks on LLM agents within the AgentDojo framework, employing both white-box (GCG) and black-box (TAP) methodologies across 80 task pairs in various domains. The results reveal that black-box optimization significantly outperforms gradient-based methods due to GCG's instability under limited compute resources, while the effectiveness of TAP is influenced by the attacker's model capabilities and safety tuning. Notably, task-universal attacks show promising transferability to unseen tasks, yet smaller model optimizations fail to generalize to advanced models like GPT-5, underscoring the model-dependent nature of this threat.
Black-box prompt injection attacks can significantly outperform traditional methods, revealing a critical vulnerability in LLM agents that could be exploited in real-world applications.
Indirect prompt injection poses a critical threat to LLM agents that interact with untrusted external data, yet automated attack methods--proven effective for jailbreaking--remain underexplored in realistic agentic settings. We present a comprehensive empirical evaluation of automated prompt injection attacks against LLM agents, adapting both white-box (GCG) and black-box (TAP) methods to the agentic setting within the AgentDojo framework. We evaluate across 80 task pairs spanning four domains and multiple models, and find that black-box optimization substantially outperforms gradient-based methods, a gap we attribute to GCG's optimization instability under reasonable compute budgets. We also find that TAP's effectiveness depends on the attacker model, as both general capability and safety tuning affect attack success--stronger models produce more effective injections, while safety-tuned attackers can refuse to generate adversarial prompts. Task-universal attacks transfer effectively to unseen tasks and out-of-distribution domains, but attacks optimized on smaller open-source models do not transfer to frontier models like GPT-5. These findings highlight automated prompt injection as a credible but model-dependent threat, with significant barriers remaining for model-agnostic exploitation.
ETH