Search papers, labs, and topics across Lattice.
The paper introduces BenGER, a new benchmark dataset for evaluating LLMs on subsumption-based legal reasoning in German law, comprising 596 exam-style case tasks and 531 doctrinal reasoning tasks. They evaluate 12 LLMs, including closed and open-weight models, using both automatic metrics and a novel "LLM-as-a-Judge" framework validated against human grading. Results show closed-source models perform best, and human-AI co-creation significantly improves upon unaided human performance, while the LLM-as-a-Judge framework maintains high agreement with human reviewers.
LLMs can now grade legal reasoning with human-level consistency, paving the way for automated assessment and feedback in legal education.
We introduce the BenGER (Benchmark for German Law) dataset for evaluating LLM systems on subsumption-based legal reasoning in German law. The BenGER dataset consists of three components: 596 exam-style free-text legal case tasks across multiple levels of legal education and 531 short doctrinal reasoning tasks. We evaluate 12 contemporary LLM systems -- closed flagship, efficiency-oriented, and open-weight -- across automatic and judge-based metrics. On a controlled validation subset of timed human-written solutions under both unaided and human--AI co-creation conditions, we contextualise model performance against these human baselines. We introduce a rubric-aligned LLM-as-a-Judge framework cross-validated against a multi-rater human-grading protocol (three blind reviews plus one author-informed creator review per solution). Our results show that replacing a blind human reviewer with the LLM judge degrades agreement with the full human pool no more than removing that reviewer altogether (Calderon r=0.96 vs.~r=0.96, matched n=30), that closed-flagship systems lead the leaderboard across all corpora, and that human--AI co-creation substantially outperforms unaided human work.
