Search papers, labs, and topics across Lattice.
ActuBench is a multi-agent LLM pipeline designed to automatically generate and evaluate actuarial assessment items, with agents specialized for drafting, distractor creation, verification/repair, and auxiliary tasks like summarization. The pipeline's independent verification agent significantly improves item quality through a repair loop. Evaluation of 50 LLMs on the generated benchmark reveals that open-weight models offer competitive performance and that LLM-as-judge evaluation is crucial for accurate model discrimination, especially at the high end of the performance spectrum.
Open-source LLMs running on commodity hardware can rival proprietary models on complex actuarial reasoning tasks, but only if you use an LLM judge instead of multiple-choice questions to evaluate them.
We present ActuBench, a multi-agent LLM pipeline for the automated generation and evaluation of advanced actuarial assessment items aligned with the International Actuarial Association (IAA) Education Syllabus. The pipeline separates four LLM roles by adapter: one agent drafts items, one constructs distractors, a third independently verifies both stages and drives bounded one-shot repair loops, and a cost-optimized auxiliary agent handles Wikipedia-note summarization and topic labelling. The items, per-model responses and complete leaderboard are published as a browsable web interface at https://actubench.de/en/, allowing readers and practitioners to inspect individual items without a repository checkout. We evaluate 50 language models from eight providers on two complementary benchmarks -- 100 empirically hardest multiple-choice items and 100 open-ended items scored by an LLM judge -- and report three headline findings. First, multi-agent verification is load-bearing: the independent verifier flags a majority of drafted items on first pass, most of which the one-shot repair loop resolves. Second, locally-hosted open-weights inference sits on the cost-performance Pareto front: a Gemma~4 model running on consumer hardware and a Cerebras-hosted 120B open-weights model dominate the near-zero-cost region, with the latter within one item of the top of the leaderboard. Third, MCQ and LLM-as-Judge rankings differ meaningfully: the MCQ scaffold inflates the performance ceiling, and Judge-mode evaluation is needed to discriminate at the frontier.
