Search papers, labs, and topics across Lattice.
This paper formalizes the emerging field of AI agent security by constructing an attack taxonomy for language models and developing a security-game framework to capture completeness and adversarial robustness in agentic access control. The framework unifies confidentiality, integrity, and availability, revealing a fundamental conflict between training data confidentiality and completeness in existing approaches. Finally, the paper formalizes a modular decomposition of helpfulness and harmlessness objectives and proves its soundness, enabling principled reasoning about agentic system security.
Existing AI agent permissioning schemes are hard to compare, so this paper provides a formal foundation and reveals a fundamental conflict between training data confidentiality and agent completeness.
Recent progress in (Large) Language Models (LMs) has enabled the development of autonomous LM-based agents capable of executing complex tasks with minimal supervision. These agents have started to be integrated into systems with significant autonomy and authority. The security community has been studying their security. One emerging direction to mitigate security risks is to constrain agent behaviours via access control and permissioning mechanisms. Existing permissioning proposals, however, remain difficult to compare due to the absence of a shared formal foundation. This work provides such a foundation. We first systematize the landscape by constructing an attack taxonomy tailored to language models, the computational primitives of agentic systems. We then develop a formal treatment of agentic access control by defining an AIOracle algorithmically and introducing a security-game framework that captures completeness (in the absence of an adversary) and adversarial robustness. Our security game unifies confidentiality, integrity, and availability within a single model. Using this framework, we show that existing approaches to confidentiality of training data fundamentally conflict with completeness. Finally, we formalize a modular decomposition of helpfulness and harmlessness objectives and prove its soundness, in order to enable principled reasoning about the security of agentic system designs. Our studies suggests that if we were to design a secure system with measurable security, then we might want to use a modular approach to break the problem into sub-problems and let the composition on different modules complete the design. Our studies show that this natural approach with the relevant formalism is needed to prove security reductions.
UW