Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
Multi-Legal-Bench is introduced as a novel cross-jurisdictional legal benchmark evaluating LLMs on five legal tasks across six countries with diverse languages and legal traditions. The benchmark uses structured metadata from national court registries to create a sparse task-jurisdiction matrix, enabling direct comparison of LLM performance across different legal systems. Evaluation of 11 LLMs reveals task-dependent few-shot learning effects replicate across jurisdictions, cross-lingual transfer is better predicted by label-set alignment than language family, and tokenizer fertility is not a strong predictor of cross-lingual accuracy.
Forget language families – label alignment is the secret sauce for cross-lingual transfer in legal reasoning, outperforming linguistic proximity.
Legal NLP benchmarks overwhelmingly evaluate a single language or aggregate tasks that differ fundamentally across jurisdictions, making cross-lingual comparison impossible. We introduce Multi-Legal-Bench, the first cross-jurisdictional legal benchmark that evaluates identical tasks across six countries (Ukraine, France, Netherlands, Poland, Czech Republic, Lithuania), four language families, and 134 million court decisions. The benchmark defines five tasks court-type classification, judgment form classification, case-outcome prediction, legal norm extraction, and cause category prediction mapped to structured metadata from national court registries, forming a deliberately sparse 5x6 task-jurisdiction matrix (20 of 30 cells filled). We evaluate 7 frontier LLMs under zero-shot and 3-shot prompting via AWS Bedrock, with 4 additional small/medium models (3-12B) for scaling analysis. Our results reveal that: (1) task-dependent few-shot effects discovered in Ukrainian replicate across all jurisdictions; (2) no single model dominates any language rankings shift with both task and jurisdiction; (3) cross-lingual few-shot transfer does not follow language proximity: UA->FR (Romance, -2.1 pp) transfers better than UA->PL (Slavic, -13.7 pp), with label-set alignment predicting transfer quality better than language family; and (4) tokenizer fertility, despite a 2.3x spread, does not significantly predict cross-lingual accuracy (r=-0.27, p=0.14), suggesting that model architecture and pretraining data dominate tokenizer efficiency. We release all data, prompts, and model predictions.