Search papers, labs, and topics across Lattice.
The authors introduce DiscoverPhysics, a benchmark designed to evaluate LLMs' ability to discover novel physics laws in simulated environments by iteratively proposing experiments, observing trajectory data, and formulating explanations. They test 11 frontier models across 22 simulated worlds with varying physics, including screened gravity and hidden dark matter, evaluating performance based on trajectory MSE and LLM-judged explanation quality. Results show that even the strongest models struggle with worlds requiring discovery of latent structures, and that strong predictive accuracy does not guarantee good conceptual understanding.
LLMs can ace physics exams, but put them in a universe with slightly different laws and their scientific reasoning skills fall apart.
Frontier LLMs now perform strongly across a wide range of physics evaluations, but it is hard to disentangle genuine reasoning from recall of established science. We introduce DiscoverPhysics, an interactive benchmark that asks a LLM agent to discover the laws of motion of a simulated world whose physics deliberately deviates from our own. We construct 22 worlds governed by, among others, screened and fractional-power gravity, multi-species couplings, hidden dark-matter-like particles, non-coordinate-free physics, and time-varying interactions. Each world is generated on demand by an N-body simulator, for which the agent proposes several rounds of experiments, observes raw trajectory data, and ultimately submits both a natural-language explanation of the world's physics and a Python implementation of the inferred law. Because solving a world requires the agent to design informative experiments and revise its hypotheses, the benchmark probes long-horizon reasoning over an experimental history. We evaluate submissions along two complementary axes: trajectory MSE on held-out particles and an LLM-judged explanation score following an expert-written rubric assessing conceptual understanding of each world. Across eleven frontier models, we find that the strongest agents pass only half of the worlds and consistently fail on those where latent structure must be uncovered. Open-source models lag substantially behind commercial models, both in their ability to design informative experiments and in extracting conclusions from the data. We further find that good predictive accuracy does not guarantee high explanation quality and that conceptual understanding depends on hypothesis refinement through well-chosen experiments.
