Search papers, labs, and topics across Lattice.
This study introduces a novel four-stage diagnostic to evaluate the reasoning capabilities of large language models (LLMs) in unfamiliar physics frameworks, addressing the limitations of traditional benchmarks that primarily measure answer accuracy. By applying this diagnostic across three distinct physics worlds, the authors reveal that models like Claude Opus 4.7 and GPT-5.5 struggle significantly, achieving only 6 out of 15 correct responses in two worlds, while Gemini 3.1 Pro fails entirely in the third. Notably, the findings highlight a qualitative-versus-quantitative asymmetry in model performance, where LLMs reliably predict directional changes but frequently miscalculate ratios, indicating a fundamental gap in their reasoning abilities.
LLMs excel at predicting the direction of change in unfamiliar physics scenarios but falter dramatically in quantitative reasoning, revealing critical gaps in their understanding.
Current large-language-model (LLM) physics benchmarks are usually scored by answer accuracy, which cannot distinguish genuine reasoning from recall of familiar problem patterns and reveals little about where a model's reasoning breaks down. We introduce an auditable four-stage diagnostic that evaluates whether an LLM can reason inside an unfamiliar physics framework through induction, formulation, prediction, and review. The diagnostic combines locked pre-registrations, fresh sessions between stages, dual-LLM judging, and a human-audit pathway, and we apply it to three parallel physics worlds: a single-equation counterfactual world ($F=mv$), a historical framework (Aristotelian mechanics), and a four-domain counterfactual world (Decay World). Across Claude Opus 4.7, GPT-5.5, and Gemini 3.1 Pro, the three worlds yield composite PASS rates are 6/15, 6/15, and 0/15 respectively (content $\land$ structural for $F=mv$ and Aristotelian, content axis only for Decay World where the structural axis is out of scope). The most pointed empirical pattern is a qualitative-versus-quantitative asymmetry: in Decay World, models almost never predict the wrong direction of change, but frequently compute the wrong ratio by slipping back to standard-physics relations. The protocol also surfaces two methodology findings: LLM-judge reliability does not transfer across frameworks, and Stage 4 self-review is weak in every framework, with the model's own review wrongly reporting no earlier error in at least two-thirds of the trials that actually contained one. We release the full prompts, responses, verdicts, and audit records.
