Search papers, labs, and topics across Lattice.
This paper introduces ParEvalLayer, a decision-making framework that evaluates LLM-agent performance based on partial outcomes from benchmark tasks. The method addresses the limitations of relying solely on early task scores, which can misrepresent the overall performance due to incomplete evidence. The evaluation of ParEvalLayer reveals that for three public benchmarks, it can accurately match the final decisions after observing only 15% to 25% of the task outcomes, highlighting the importance of context in performance assessments.
Relying on partial evaluations can lead to misleading conclusions, but ParEvalLayer ensures accurate decision-making with as little as 25% of the data.
LLM-agent evaluations often produce task outcomes long before the full benchmark run is complete. A partial score is tempting to report, but it does not show whether the observed tasks support the same conclusion as the completed evaluation. Early tasks can omit important parts of a benchmark, running cheaper tasks first can distort the observed sample, and a rule that decides only easy pairs can appear accurate while leaving many comparisons unresolved. We introduce ParEvalLayer, a decision layer that reads paired outcomes for two agent systems and a comparison policy chosen in advance. For each partial run, it records whether the tested agent system is better by the required amount, is not better by that amount, needs more evidence, or should abstain. We evaluate ParEvalLayer by replaying completed public benchmark data as if each evaluation had stopped earlier. At each point, ParEvalLayer applies the policy using only the outcomes observed so far; if it reaches one of the two comparison judgments, we check whether that judgment matches the completed data for the same system pair. With the main comparison rule, three of the public benchmarks reach the same decision as the completed evaluation after observing only 15% to 25% of task outcomes. Other benchmarks require more task outcomes. This variation shows why a partial score alone is not enough: reports should also state the decision rule and how many comparisons remain without a decision.
