Search papers, labs, and topics across Lattice.
This paper introduces InsufficiencyBench, a novel benchmark designed to evaluate large language models (LLMs) on their ability to handle underspecified legal queries. It highlights the critical issue that existing benchmarks do not account for the omission of essential facts that can significantly influence legal outcomes. The evaluation of ten leading models reveals that none surpass an F2 score of 0.46 in identifying missing elements, indicating a widespread inability to appropriately manage incomplete legal queries while providing accurate responses to fully specified ones.
No current LLM can accurately identify missing legal information in user queries, with all evaluated models struggling to balance responses to both deficient and complete questions.
Legal AI systems are increasingly used to answer legal questions, yet existing benchmarks assume queries arrive fully specified. In practice, users omit facts that materially determine the legal outcome. We introduce InsufficiencyBench, the first legal benchmark targeting query-side insufficiency: whether a model recognizes when a query lacks legally material information, identifies what is missing, and refrains from premature conclusions. We formalize a taxonomy of eight canonical missing-element categories across three structural failure modes---switch, gating, and fatal prerequisite--- and construct 202 benchmark items (58 base queries, 144 deficient variants) spanning six legal domains and 24 US jurisdictions and annotated by practising attorneys. Evaluating ten frontier models, we find that no model exceeds F2 = 0.46 on missing-element identification and that the median recall is 0.44. Models either hedge indiscriminately or answer silently under fabricated presumptions. No model both identifies and qualifies responses to deficient queries while directly addressing complete ones.
