Search papers, labs, and topics across Lattice.
The paper introduces POLARIS, a framework for systematically generating safety tests for LLMs from natural language policy specifications. POLARIS compiles policies into First-Order Logic, constructs a Semantic Policy Graph encoding violation scenarios, and generates natural language test queries by exploring this graph. Experiments show POLARIS achieves higher policy coverage and attack success compared to existing methods, offering a more principled and automated approach to LLM safety testing.
Forget red-teaming, POLARIS automatically turns safety policies into attack strategies, finding more LLM vulnerabilities with verifiable traceability.
The widespread integration of Large Language Models (LLMs) necessitates rigorous and systematic safety evaluation. Existing paradigms either rely on constructed benchmarks to assess safety from predefined perspectives, or employ dynamic red-teaming to probe potential vulnerabilities. While effective, these approaches face challenges, as they depend heavily on expert domain knowledge, offer limited systematic guarantees, and are vulnerable to rapid obsolescence. To address these limitations, we introduce a novel framework POLARIS that brings the rigor of specification-based software testing to AI safety. POLARIS first compiles unstructured natural-language policies into First-Order Logic (FOL) representations, establishing a traceable link between high-level rules and concrete test cases. This formalization enables the construction of a Semantic Policy Graph, where complex policy violation scenarios are encoded as traversable paths. By systematically exploring this graph, POLARIS uncovers compositional violation patterns, which are then instantiated into executable natural-language test queries, enabling coverage-driven and reproducible safety testing. Experiments demonstrate that POLARIS achieves higher policy coverage and attack success counts compared to established baselines. Crucially, by bridging formal methods and AI safety, POLARIS provides a principled, automated approach to ensuring LLMs adhere to safety-critical policies with verifiable traceability. We release our code at https://github.com/huac-lxy/POLARIS.
NUS