Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
To evaluate whether AI scientific agents can uncover latent generative processes rather than merely curve-fitting observational data, the authors establish MechBench to explicitly disentangle phenomenal-law recovery from underlying mechanism discovery. Across leading agent setups, mechanism discovery dramatically lags phenomenal-law fitting—for instance, Codex with GPT-5.6-sol achieves 35.00% phenomenal accuracy but only 13.75% mechanism accuracy, failing mechanism identification in 64.29% of instances where it solved the phenomenal law. This generalization gap widens under mutated, non-textbook variants and persists even when agents are directly provided with the ground-truth phenomenal law, exposing a fundamental bottleneck in automated scientific reasoning.
AI scientific agents frequently solve empirical equations for entirely the wrong reasons, failing to identify the correct generative mechanism in over 64% of the cases where they successfully recover the observable phenomenal law.
Scientific discovery requires not only recovering mathematical laws that describe observable behavior, but also identifying the mechanisms that generate them. Existing benchmarks for symbolic regression and scientific agents primarily evaluate phenomenal-law recovery, leaving mechanism discovery largely untested. We introduce MechBench, a benchmark that explicitly separates these two capabilities. Each task is defined by a mechanistic model, a structured set of scientifically meaningful relations whose joint consequences entail an observable phenomenal law, while agents receive only observational data and scientific context. We evaluate mechanism recovery through mechanism probes, which query internal scientific consequences that cannot be inferred from the phenomenal law alone. To reduce reliance on memorized textbook mechanisms, we construct unfamiliar variants through controlled, scientifically interpretable mutations of canonical mechanisms, and screen for mechanistic indistinguishability to exclude ambiguous instances admitting comparable competing mechanisms. Experiments across representative scientific agents reveal a substantial phenomenal--mechanism recovery gap: for Codex with GPT-5.6-sol, phenomenal-law accuracy reaches 35.00% on the Core-set while mechanism accuracy is only 13.75%, with mechanism recovery failing in 64.29% of cases where the phenomenal law is correctly recovered. The gap widens as mechanisms become increasingly mutated, and even providing the correct phenomenal law leaves mechanism recovery below 50%. These results reveal a substantial generalization gap in mechanistic reasoning and establish mechanism discovery as a distinct challenge beyond recovering observable scientific laws.
Tsinghua AI