Search papers, labs, and topics across Lattice.
This paper introduces an autoformalization pipeline that converts agent instructions and natural language policy documents into formally verified policies using a generator-critic loop based on large language models (LLMs). This approach addresses the limitations of existing methods that either lack formal guarantees or do not scale effectively, thereby enhancing agent safety in high-stakes environments. The results show that the autoformalized policies significantly outperform previous hand-coded symbolic enforcement in terms of coverage on the MedAgentBench benchmark.
Autoformalization using LLMs can achieve formal policy enforcement that scales beyond traditional hand-coded methods, dramatically improving agent safety in critical applications.
Agent safety in high-stakes domains requires formal policy enforcement, but most existing approaches either rely on probabilistic guardrails (fine-tuned classifiers, prompt-based steering) that offer no formal guarantees, or on hand-coded symbolic enforcement that does not scale to the breadth of real policy specifications. We present an autoformalization pipeline that translates agent prompts, MCP tool descriptions, and natural language policy documents into formally verified policies using an LLM-based generator-critic loop. The resulting policies are written in the Cedar Policy Language. On the MedAgentBench benchmark, our autoformalized policies cover substantially more of the source natural-language specification than the hand-coded symbolic enforcement in prior work.
