Search papers, labs, and topics across Lattice.
This paper introduces an agentic framework that leverages knowledge graphs (KGs) constructed from AI policy documents to enhance LLM-based policy compliance reasoning. The framework builds KGs from three AI risk-related policies using two ontology schemas and evaluates five LLMs on 42 policy QA tasks. Results show that KG augmentation consistently improves LLM performance across various reasoning types, and surprisingly, an LLM-discovered schema performs comparably to a formal ontology.
Forget hand-crafted ontologies: LLMs armed with knowledge graphs built from policy documents can reason about AI compliance just as well (or better!) using schemas they invent themselves.
The risks posed by AI features are increasing as they are rapidly integrated into software applications. In response, regulations and standards for safe and secure AI have been proposed. In this paper, we present an agentic framework that constructs knowledge graphs (KGs) from AI policy documents and retrieves policy-relevant information to answer questions. We build KGs from three AI risk-related polices under two ontology schemas, and then evaluate five LLMs on 42 policy QA tasks spanning six reasoning types, from entity lookup to cross-policy inference, using both heuristic scoring and an LLM-as-judge. KG augmentation improves scores for all five models, and an open, LLM-discovered schema matches or exceeds the formal ontology.
