Search papers, labs, and topics across Lattice.
This study investigates the ability of language models to verify the support of legal citations at the proposition level, particularly focusing on the distinction between correct case citations and pinpoint page references. Through controlled perturbations of real legal citations from two corpora, the authors evaluate fourteen model configurations, revealing that while models excel at identifying wrong-case citations (93-100% accuracy), they struggle significantly with wrong-pinpoint citations, achieving only 37-61% accuracy on court opinions. The findings highlight a critical gap in current models' capabilities, as they often conflate topic recognition with citation support verification, leading to substantial errors in legal contexts.
Language models may ace wrong-case citation detection but fail to verify pinpoint accuracy, missing 40% of critical errors even in advanced configurations.
In 2023, a New York judge sanctioned two attorneys in Mata v. Avianca for filing a brief with hallucinated citations generated by ChatGPT. Such failures are largely caught by database lookups; the harder problem is detecting citations that point to real cases but do not support the propositions for which they are offered -- a failure mode that existing evaluations of LLMs for legal use cases largely overlook. In this paper, we study proposition-level citation support verification through controlled perturbations of real legal citations obtained from two legal corpora, either replacing the cited case or changing only the pinpoint page within the same case. We evaluate fourteen model configurations on the resulting examples. Models catch 93-100% of wrong-case corruptions. They catch only 37-61% of wrong-pinpoint corruptions on court opinions and 52-83% on legal briefs. When models fail to catch wrong-pinpoint corruptions, they accept the citation based on topical overlap rather than page-level support. Scale and extended reasoning narrow the gap but do not close it: GPT-5.4 with high reasoning effort still misses 40% of pinpoint mismatches on court opinions and 18% on briefs. Prompting the model to verify support at the cited page improves recall, but it also raises the false positive rate. Recognizing the right legal topic and verifying support for the cited proposition are distinct capabilities, and current models conflate them.
