Search papers, labs, and topics across Lattice.
The paper introduces BenGER, an open-source web platform designed to streamline the benchmarking of LLMs on German legal tasks. It integrates task creation, collaborative annotation, model execution, and evaluation using a variety of metrics. The platform supports multi-organization projects with role-based access control and provides formative feedback to annotators, enhancing transparency and reproducibility in legal LLM evaluation.
Finally, a unified platform lets legal experts collaboratively benchmark LLMs on German law, even without coding skills.
Evaluating large language models (LLMs) for legal reasoning requires workflows that span task design, expert annotation, model execution, and metric-based evaluation. In practice, these steps are split across platforms and scripts, limiting transparency, reproducibility, and participation by non-technical legal experts. We present the BenGER (Benchmark for German Law) framework, an open-source web platform that integrates task creation, collaborative annotation, configurable LLM runs, and evaluation with lexical, semantic, factual, and judge-based metrics. BenGER supports multi-organization projects with tenant isolation and role-based access control, and can optionally provide formative, reference-grounded feedback to annotators. We will demonstrate a live deployment showing end-to-end benchmark creation and analysis.
