Search papers, labs, and topics across Lattice.
This study investigates the effectiveness of partial evaluations in agent benchmarks by analyzing task-level records from SWE-bench, AppWorld, and tau-bench. The authors find that the required task fraction for a partial evaluation to yield valid pairwise conclusions varies significantly across benchmarks, with AppWorld needing only 15% of tasks, while SWE-bench Lite fails to meet the necessary criteria even at 95%. The results highlight the importance of transparency in reporting partial evaluations, emphasizing the need for clarity on performance thresholds, task selection, and coverage rules.
Partial evaluations can mislead if not carefully calibrated, with required task fractions varying dramatically across benchmarks—15% for AppWorld but 95% for SWE-bench Lite.
Agent benchmarks often compare two agents after all tasks have run, but costly evaluations make partial runs tempting. A task fraction alone does not show whether a partial run supports the same pairwise conclusion as the completed benchmark. We study this question by replaying completed public task-level records from SWE-bench, AppWorld, and tau-bench. A partial budget counts as enough only when it supports the completed benchmark's decision, covers required task groups, and leaves no more than a target fraction of comparisons unresolved. The required task fraction varies sharply. At the strict 0 percentage point threshold on a 5 percentage point budget grid, AppWorld first meets all targets at 15 percent, tau-bench at 25 percent, and SWE-bench Verified at 90 percent; SWE-bench Lite does not meet all targets by 95 percent under the primary coverage rule. Partial-evaluation reports should state how much one agent must outperform another, how tasks are selected, what coverage rule is required, what decision rule is used, and how many comparisons may remain unresolved.
