Search papers, labs, and topics across Lattice.
This paper introduces AcquaBench, a framework for auditing success provenance in agent evaluations, which identifies whether an agent's success is due to intended reasoning or merely acquiring the correct answer. By employing a matched value substitution method across four standardized surfaces, the study reveals that the availability of the correct target significantly influences success scores, with GOLD scores exceeding SHAM by up to 25.9 percentage points. The findings highlight the importance of distinguishing between correctness and the underlying reasoning process, suggesting that agent benchmarks should report both success and the supporting information state.
Success in agent evaluations often masks the true reasoning process, with a new method revealing that access to the correct target can inflate success scores by over 25 percentage points.
A correct answer can conceal why an agent succeeded. Once agents change their information state during evaluation, correctness no longer distinguishes intended reasoning from answer acquisition. Outcome evidence and exposure detection do not establish whether success depended on an acquired target; we call this missing evaluation object success provenance. AcquaBench audits it through matched CLEAN, GOLD, and SHAM value substitution on four standardized surfaces with joint qid-clustered analysis. CLEAN retains benchmark-authorized information. GOLD makes the correct target available. SHAM preserves source structure and exposure opportunity but substitutes a matched incorrect value. GOLD minus CLEAN measures the total score response to correct-target availability; GOLD minus SHAM tests whether that response tracks target correctness beyond matched source exposure. In D0, GOLD exceeds SHAM by 19.1 to 25.9 percentage points, showing that success follows the correct value. In D2, GOLD still exceeds SHAM under distributed sufficiency while coloc no longer transfers as a high-score marker, with AUROC 0.376 and 0.142. Behavioral dependence can thus persist beyond this probe's intended observation unit. In model comparison, a supported 5.0-point CLEAN score gap compresses to a raw GOLD difference of -0.6 points without establishing rank inversion. Agent benchmarks should report success together with whether the evaluated information state supported it.
