Search papers, labs, and topics across Lattice.
Existing legal AI benchmarks rely on single-turn, outcome-level metrics that fail to detect cascading tool-calling and reasoning errors along multi-step agent trajectories. To diagnose these failures, the authors construct LexAgentHallu, a 3,414-instance trajectory benchmark spanning 17 legal categories and 6 task types annotated under a hierarchical 27-subclass taxonomy of substantive and procedural errors. Evaluating 18 frontier and open-source agents reveals a prevalent "Right-Answer-Wrong-Reason" phenomenon and demonstrates that agentic hallucinations cluster into distinct framework- and domain-specific failure profiles rather than occurring randomly.
Legal agents frequently reach the correct legal outcome through entirely fabricated citations and broken tool-use trajectories, revealing a deceptive "Right-Answer-Wrong-Reason" failure mode invisible to standard benchmarks.
As large language models are increasingly deployed as tool-augmented legal agents, they introduce agentic hallucinations where tool-call and reasoning errors cascade into fabricated holdings and miscited authority. However, existing legal benchmarks evaluate only single-turn QA with outcome-level metrics, while agentic hallucination benchmarks lack legal-specific diagnostic capability. Neither answers to what extent and how a legal agent hallucinates along its trajectory. To address these limitations, we introduce LexAgentHallu, a legal agentic hallucination benchmark designed to evaluate to what extent and how legal agents fail along multi-step trajectories. Built through a four-stage expert-in-the-loop pipeline, LexAgentHallu contains 3414 instances across 17 legal categories and 6 task types. Each instance is annotated under a dual-layer hallucination taxonomy of 7 high-level categories and 27 fine-grained subclasses, covering both substantive errors and agent-procedural failures. We further design fine-grained metrics that quantify to what extent and localize how each failure occurs along an agent's execution path. Our evaluation across 18 proprietary and open-source agents uncovers a Right-Answer-Wrong-Reason effect and reveals that hallucination subclasses cluster rather than scatter, forming distinct agentic framework, legal task, and category profiles. These findings, invisible to outcome-level evaluation, validate the diagnostic power of LexAgentHallu for evaluating agentic hallucination in law.
