Track the frontier, from condensed matter to cosmology.
Search papers, labs, and topics across Lattice.
This paper introduces Selection-Aware Semantic Stress Testing (SASST), a novel framework that evaluates interactive agents by learning task reweighting from pre-execution features and validating findings on separate confirmation tasks. The method addresses the issue of biased conclusions drawn from a single benchmark by ensuring that evaluations are independent and robust, thereby enhancing the reliability of agent assessments. Key results reveal that a previously identified advantage in agent performance dissipates upon confirmation testing, highlighting the importance of rigorous validation in agent evaluation protocols.
A previously observed performance gain in interactive agents disappears under rigorous confirmation testing, challenging the reliability of existing evaluation benchmarks.
Agent evaluations often use one benchmark to choose a workflow and then search for task types where its advantage weakens, so both conclusions are selected from the same data. We introduce Selection-Aware Semantic Stress Testing (\SASST{}), which learns a task reweighting from pre-execution features on discovery tasks and evaluates the same paired comparison on separate confirmation tasks. The protocol checks support and stability, uses joint bounds for all planned claims, and can return no claim. We prove conditional asymptotic validity under stated cluster assumptions. A forty-cluster audit finds Gaussian undercoverage and conservative Bonferroni $t$ bounds. In one 480-episode $τ$-bench study, a $3.75$ point discovery gain vanished on confirmation. A second-model study likewise confirmed neither a workflow benefit nor a stable stress rule.