Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This study introduces a scalable pipeline for generating high-quality rubrics for evaluating financial report quality without the need for human experts. By synthesizing 14,450 candidate rubrics from model-generated reports and validating their effectiveness against human evaluations, the authors demonstrate that LLM-based judgments can achieve a remarkable 98.67% agreement with human experts. The resulting consensus-derived gold rubrics, totaling 2,600, enable clear differentiation among 10 deep research systems, showcasing the potential for automated, large-scale benchmark evaluations.
LLMs can replace human experts in evaluating financial report quality, achieving near-perfect agreement in rubric judgments.
Deep research agents are increasingly used to produce long-form financial reports, yet large-scale evaluation remains bottlenecked by the need for human experts to define and execute high-quality rubrics. We address this problem by proposing a scalable pipeline for generating high-quality rubrics without human experts in the final loop. We build a financial deep research benchmark from 104 real-world user queries and automatically synthesize 14,450 query-specific candidate rubrics from model-generated reports. To justify removing human experts from rubric execution, we compare rubric judgments from three human experts with those from a three-LLM judge panel on a sampled subset, and show that LLM-based evaluation is sufficiently consistent with human evaluation to replace it for large-scale rubric screening, including 98.67\% label-level agreement on jointly unanimous items. We then derive consensus-derived gold rubrics through two filters: a strict consistency filter, which keeps a rubric only if the three LLM judges unanimously agree on every report under the same query, and a distinguishability filter, which keeps a rubric only if it assigns at least one majority-yes and at least one majority-no label across the evaluated systems. This process retains 3,687 consistency-passed rubrics, of which 2,600 remain distinguishable and form the final set of consensus-derived gold rubrics. Using this final rubric set, we obtain clearly differentiated rankings across 10 deep research systems, with item-level pass rates ranging from 58.58\% to 22.23\%. More broadly, because the pipeline removes human-expert execution from rubric generation and evaluation, it is naturally scalable for benchmark evaluation, automatic system comparison, and future studies of evaluation-driven system improvement.