Search papers, labs, and topics across Lattice.
This paper introduces a hybrid symbolic approach that combines formal logic representation of policies with the fact extraction capabilities of language models and an answer set solver for reasoning. The method significantly enhances the interpretability and robustness of responses to natural language queries related to rules, outperforming traditional policy-as-prompt and policy-as-code methods with a tenfold reduction in token usage. The findings highlight the effectiveness of structured reasoning in conjunction with generative models for making accurate decisions based on objective criteria.
Separating extraction from reasoning can achieve ten times fewer tokens while enhancing accuracy and robustness in AI responses to complex policy queries.
In many practical applications of generative AI systems, from tax rules to airline baggage allowance, responses to natural language queries must respect written policies or rules. We present a hybrid symbolic approach that expresses policies in formal logic and at inference time exploits the representation power of language models for fact extraction to ground predicates, and an answer set solver for reasoning such that responses are interpretable, auditable, and as we show, accurate and robust under input perturbations. Specifically, we show this separation of extraction and reasoning steps outperforms policy-as-prompt and policy-as-code methods in most cases with ~10x reduction in token usage. The results point to the value of structured reasoning and symbolic solvers in conjunction with generative models to make robust decisions involving objective criteria.
