Search papers, labs, and topics across Lattice.
The paper introduces GrantBox, a security evaluation sandbox that integrates real-world tools to assess LLM agent privilege usage under prompt injection attacks. GrantBox allows agents to invoke genuine privileges, enabling a more realistic security evaluation compared to existing benchmarks with pre-coded tools. Experiments reveal that LLMs, despite exhibiting basic security awareness, are highly vulnerable to sophisticated attacks, achieving an average attack success rate of 84.80% in crafted scenarios.
LLM agents controlling real-world tools are alarmingly easy to manipulate, with an 85% success rate for privilege escalation attacks, despite exhibiting basic security awareness.
Equipping LLM agents with real-world tools can substantially improve productivity. However, granting agents autonomy over tool use also transfers the associated privileges to both the agent and the underlying LLM. Improper privilege usage may lead to serious consequences, including information leakage and infrastructure damage. While several benchmarks have been built to study agents'security, they often rely on pre-coded tools and restricted interaction patterns. Such crafted environments differ substantially from the real-world, making it hard to assess agents'security capabilities in critical privilege control and usage. Therefore, we propose GrantBox, a security evaluation sandbox for analyzing agent privilege usage. GrantBox automatically integrates real-world tools and allows LLM agents to invoke genuine privileges, enabling the evaluation of privilege usage under prompt injection attacks. Our results indicate that while LLMs exhibit basic security awareness and can block some direct attacks, they remain vulnerable to more sophisticated attacks, resulting in an average attack success rate of 84.80% in carefully crafted scenarios.
Tsinghua AI