Search papers, labs, and topics across Lattice.
This study introduces TW-LegalBench, a benchmark designed to assess the performance of large language models (LLMs) on Taiwanese legal reasoning using a comprehensive corpus of official legal materials. By evaluating 13 LLMs across multiple task types鈥攊ncluding over 16,000 multiple-choice questions and legal judgment predictions鈥攖he researchers found that while top models can pass qualification exams for lawyers, they significantly underperform on tasks requiring precise legal citations. The results underscore the limitations of LLMs in generating reliable legal text, despite their competitive performance on qualification assessments.
LLMs can pass Taiwanese lawyer qualification exams but struggle with precise legal citations, revealing critical gaps in their legal reasoning capabilities.
Large language models (LLMs) have shown impressive capabilities across diverse tasks, yet their performance on jurisdiction-specific legal reasoning remains underexplored. We present TW-LegalBench that utilizes Taiwanese legal system's rich official corpus open to the public to fill the gap in evaluating LLMs on Taiwanese law, among common-law benchmarks that focus on English sources and civil-law benchmarks focusing on sources of Simplified Chinese. TW-LegalBench comprises three task types: (1) over 16,000 multiple-choice questions (MCQs) across five years of official examinations in 18 professional domains; (2) 117 open-ended essay questions (OEQs) from examinations for legal professionals with official scoring rubrics; and (3) more than 14,000 legal judgment prediction (LJP) instances covering hundreds of crime categories. We evaluate 13 LLMs using accuracy for MCQs, a decomposed LLM-as-Judge framework based on the scoring rubric points for OEQs, and metrics for sentencing accuracy and statute citation for LJP. Our results reveal that top-performing models exceed the passing threshold for qualified lawyers (passing rate: 11%) but fall short of that for judges and prosecutors (passing rate: 1~2%). For LJP, while models demonstrate reasonable verdict type accuracy and sentence prediction capability, they struggle to cite exact legal articles. These findings highlight that reliable legal text generation remains challenging for LLMs, even though their performance on qualification examinations approaches human level.
NVIDIA