Search papers, labs, and topics across Lattice.
This paper introduces CrossLex, a benchmark designed to evaluate large language models (LLMs) on cross-jurisdictional legal reasoning by utilizing a dataset grounded in authoritative legal sources from China, California, and Germany. The benchmark consists of 6,149 instances organized into 385 fact groups, addressing various legal issues and enabling the assessment of models on tasks that distinguish between single-jurisdiction reasoning and cross-jurisdictional comparisons. Experimental results reveal that while LLMs can often answer legal questions correctly, they significantly struggle with providing accurate legal citations across different jurisdictions, highlighting a critical gap in their reasoning capabilities.
LLMs may excel at answering legal questions, but they falter when it comes to accurately citing laws across jurisdictions, revealing a major flaw in their legal reasoning abilities.
Legal reasoning is inherently jurisdiction-dependent: the same facts can call for different legal rules and yield different conclusions across legal systems. Yet existing benchmarks rarely evaluate whether large language models (LLMs) can recognize such jurisdiction-specific variation, especially when identical fact patterns lead to divergent legal outcomes.We introduce CrossLex, a same-fact, legal-source-grounded benchmark for evaluating cross-jurisdictional legal reasoning in LLMs across three jurisdictions: China, California, and Germany. Built from authoritative legal sources, CrossLex aligns 55 legal issues spanning contract, consumer, criminal, family, and labor law, and constructs jurisdiction-aligned questions paired with answers and supporting citations. In total, CrossLex contains 6,149 instances organized into 385 fact groups, with all legal issues, answers, and cited authorities reviewed by legal professionals.To disentangle basic legal knowledge from cross-jurisdictional reasoning, CrossLex defines three complementary tasks: single-jurisdiction reasoning (T1), joint cross-jurisdictional comparison (T2), and fine-grained cross-jurisdictional evaluation (T3). We further propose Grounded Joint, a metric that jointly assesses answer correctness and legal-source grounding, and provide a unified evaluation for streamlined benchmarking. Extensive experiments on representative LLMs show that, although current models can often answer legal questions correctly, they struggle to provide accurate cross-jurisdictional legal citations.We hope that CrossLex will facilitate future research on source-grounded cross-jurisdictional legal reasoning.
