Search papers, labs, and topics across Lattice.
This paper introduces LegalHalluLens, an auditing framework designed to address the high hallucination rates of AI systems in legal workflows, which average around 52%. By employing typed hallucination profiles, a Risk Direction Index (RDI), and a calibrated multi-agent debate pipeline, the framework reveals significant discrepancies in error types and directions that traditional metrics overlook. The results demonstrate a 45% reduction in fabricated detections while highlighting a critical 38-40 percentage point gap in performance across different claim categories, underscoring the need for nuanced evaluation in legal AI applications.
Traditional metrics mask critical error patterns in legal AI, but LegalHalluLens reveals a staggering 40-point gap in performance across claim types, offering a clearer path to trustworthy deployment.
AI systems deployed in legal workflows hallucinate at rates that aggregate metrics report at ~52%, but this average conceals where errors concentrate and in which direction they run, leaving compliance officers without an actionable signal for trustworthy deployment. We present LegalHalluLens, an auditing framework with three components: typed hallucination profiles across four legally-motivated claim categories (numeric, temporal, obligation/entitlement, factual) over CUAD (Hendrycks et al., 2021); a Risk Direction Index (RDI) that reduces omission-versus-invention bias to a single deployment-comparable scalar; and a typed debate pipeline calibrated to both magnitudes and directions. Across 510 contracts and 249,252 clause-level instances we measure a within-model gap of approximately 38-40 pp between obligation/numeric and temporal claims that aggregate reporting hides, and show that two systems with matched 52% rates can carry opposite RDIs. The debate pipeline reduces fabricated detections by 45% with per-category gains tracking the diagnosis, matching commercial APIs with a substantially smaller backbone (4B active parameters). Typed profiles and RDI surface failure modes that aggregate metrics hide; we further show these diagnostics serve as calibration inputs for multi-agent debate pipelines, where Skeptic challenges and asymmetric gates targeted at measured failure modes outperform generically-tuned debate. The framework supports direction-aware procurement, accountability, and agent design for legal AI deployed in the wild.
