Search papers, labs, and topics across Lattice.
This study introduces PhysElite, a comprehensive bilingual multimodal benchmark designed to evaluate large language models (LLMs) on Olympiad-level physics problems, addressing the limitations of existing benchmarks in both difficulty and coverage. The benchmark comprises 11,586 high-difficulty problems, each accompanied by visual diagrams and detailed bilingual solution derivations, enabling a thorough assessment of model capabilities. Benchmarking 18 different LLMs revealed that even the most advanced models achieved only 33.7% accuracy, highlighting significant gaps in their reasoning abilities for complex physics tasks.
Even the best LLMs struggle with Olympiad-level physics, achieving only 33.7% accuracy on a new benchmark that challenges their reasoning capabilities.
Understanding how (multimodal) large language models perform on physics problems requires benchmarks that reflect the difficulty and breadth of expert-level physical reasoning. Existing physics benchmarks remain limited in the following two important ways: (1) short of high-difficulty datasets, and (2) lack of comprehensive coverage of visual forms, knowledge points, and step-by-step solution processes. As a result, model performance on current datasets may not be fully representative of their ability to solve complex physics problems. To address these issues, we present PhysElite, a large-scale bilingual multimodal benchmark for Olympiad-level physics reasoning. PhysElite contains 11,586 Olympiad-tier problems. For each problem, we provide corresponding visual diagrams, step-by-step bilingual Chinese-English solution derivations, and the final answer. We benchmark 18 open-source and closed-source MLLMs, and find that even the strongest model reaches only 33.7% answer accuracy. We additionally conduct step-level process evaluation to diagnose where models fail in the reasoning chain. Our datasets are released at https://huggingface.co/datasets/physelite/PhysElite.
