Search papers, labs, and topics across Lattice.
This position paper critiques the evaluation of AI scientist systems, arguing that solely focusing on final outcomes neglects the importance of mechanism fidelity and epistemic honesty. Through an analysis of 28 episodes involving a coding agent in a Geant4 simulation, the authors identify a phenomenon termed Correct Answer, Wrong Mechanism (CAWM), where agents produce correct results through flawed reasoning that fails under varying conditions. The study highlights the need for separate assessments of task outcomes and underlying mechanisms, proposing a lightweight verification test to improve trust in AI-generated scientific claims.
AI agents can yield correct results while relying on fundamentally flawed reasoning, raising critical questions about their reliability as scientific co-authors.
AI scientist systems are described as tools, coauthors, or founders, but we evaluate them as if only the final answer matters. This position paper argues that outcome-only evaluation is insufficient, and that task outcome, mechanism fidelity, and epistemic honesty must be measured separately. Our evidence comes from 28 episodes of a coding agent attempting to rediscover a known particle identification observable in a Geant4 simulation, including an 8-episode probe across two additional frontier models. In 4/20 primary-model and 3/8 cross-model episodes, agents reach right-looking results through incorrect reasoning that breaks when conditions change, which we call Correct Answer, Wrong Mechanism (CAWM). Honesty and mechanism fidelity dissociate within a single agent trajectory. When given a partially misleading prior, all five agents reject the false component on evidence, yet one defends its chosen observable with physics inconsistent with its own data. In the simulation-based discovery setting studied here, coding agents prove reliable tools but unreliable scientific co-authors for open-ended claim-making, where co-author trust requires mechanism-fidelity verification they do not reliably self-apply. The failure is detectable, and we propose a lightweight test. A one-step regime-shift check needs only the agent's claim and flags the over-generalized cases. A companion recomputation flags the remaining cases when the correct observable is known. Together, these checks flag every CAWM case in this study.
