Search papers, labs, and topics across Lattice.
This paper introduces Deontic Agentic Reasoning (DAR), an innovative framework that enables large language models (LLMs) to interact with legal statutes dynamically to enhance deontic reasoning tasks. The study evaluates DAR across various agentic harnesses on challenging subsets of DeonticBench, revealing that while these harnesses can significantly improve performance, they also expose weaknesses in certain models, particularly in numerical reasoning tasks. The findings underscore the complexity of integrating agentic reasoning with LLMs, highlighting both the potential for advancements and the risks of performance degradation in specific contexts.
Agentic harnesses can significantly enhance LLM performance on deontic reasoning tasks, but not without introducing risks of degradation in numerical accuracy.
Deontic reasoning is the task of answering questions by applying explicit rules and policies to case-specific facts, for example computing tax liability under a statute or determining the outcome of an immigration appeal. A key technical challenge for LLM-based deontic reasoning is that the relevant ruleset can be long and cross-referenced, so models may still fail to locate the rules needed for a particular reasoning step. We introduce Deontic Agentic Reasoning (DAR), an agentic reasoning setup in which the model interacts with the statutes on demand. We evaluate DAR under multiple harnesses on hard subsets of DeonticBench. Across these settings, we find that agentic harnesses can push the frontier on deontic reasoning tasks, but improvements are not uniform: weaker models often degrade on numerical tasks while consuming far more tokens.
Apple ML