Search papers, labs, and topics across Lattice.
This study introduces SCILAWS-BENCH, a benchmark designed to evaluate the ability of large language models (LLMs) to discover scientific laws from real data and published research. By incorporating two distinct settings鈥擲CILAWS-REAL for fixed observations and SCILAWS-PARALLEL for active querying鈥攖he authors assess both predictive fit and scientific validity, revealing that LLMs often struggle with memorization and face a selection bottleneck in law discovery. The findings highlight a divergence between predictive accuracy and scientific validity, emphasizing the complexity of evaluating LLMs in scientific contexts.
LLMs may excel at predicting outcomes but often fail to grasp the underlying scientific validity of the laws they propose, revealing critical gaps in their discovery capabilities.
Scientific equation discovery has long been central to scientific progress, proceeding through iterative cycles of hypothesis generation, observational testing, and refinement under scientific constraints. As LLM capabilities advance and their role in AI for Science expands, it remains an open problem whether they can genuinely discover scientific laws and how this ability should be evaluated. Existing evaluations, however, often either simplify discovery through synthetic settings or reuse published targets that may already be familiar to LLMs. We therefore introduce SCILAWS-BENCH, a benchmark for scientific law discovery built from published research and real scientific data. It comprises 118 problems drawn from 381 scientific papers, covering 291 candidate laws and roughly 8M real data points across six scientific disciplines. Each problem is instantiated in two complementary settings: (1) SCILAWS-REAL asks models to propose laws from fixed real observations and evaluates held-out predictive fit and scientific validity derived from the source literature, and (2) SCILAWS-PARALLEL asks models to actively query residual-calibrated worlds and recover synthesized hidden laws derived from published forms. This two-setting task design preserves each problem's scientific context while separately evaluating fixed-record law discovery and active recovery of a newly synthesized hidden law. We find that predictive fit can diverge from scientific validity, memorization shapes whether models reproduce or move beyond published formulas, and our best-of-N study reveals a selection bottleneck. Our work provides a paper-grounded benchmark and new empirical perspectives for evaluating AI for scientific discovery. Project page: https://yiyihum.github.io/SciLaws-Bench
