Search papers, labs, and topics across Lattice.
This paper introduces PhySciBench, a comprehensive benchmark designed to evaluate Large Language Model (LLM)-based agents in the physical sciences, featuring 200 expert-curated questions across various scientific tasks. Evaluations reveal that even state-of-the-art models like Gemini Deep Research only achieve 33.5% accuracy, highlighting significant deficiencies in reasoning, knowledge transfer, and self-verification. In response, the authors present DelveAgent, a multi-agent framework that enhances accuracy by up to 7.5 percentage points while reducing inference costs to one-third of the strongest baseline, underscoring the importance of specialized architectures in scientific research automation.
PhySciBench reveals that top LLMs struggle with scientific reasoning, achieving only 33.5% accuracy, while DelveAgent demonstrates a promising 7.5% improvement in performance.
Deep research agents are Large Language Model (LLM)-based systems designed for autonomous, multi-step scientific reasoning, and they hold immense potential for accelerating research in the physical sciences. However, comprehensive and in-depth evaluations of their capabilities within this domain remain lacking. To address this gap, we introduce PhySciBench, a benchmark highly relevant to physical science research, comprising 200 expert-curated questions, balanced between physics and chemistry, across six task categories that reflect real-world scientific workflows. Evaluations of state-of-the-art models and agent systems on PhySciBench reveal limited performance; even the strongest baseline, Gemini Deep Research, achieves an accuracy of only 33.5%. Analysis of failure cases identifies three recurrent deficiencies: fragility in extended reasoning chains, limited knowledge transfer across steps, and a lack of physics-grounded self-verification. Motivated by these findings, we develop DelveAgent, a modular multi-agent framework equipped with an adaptive planning loop, dual-granularity memory, and a hierarchical physics-grounded reflection mechanism. Across four scientific benchmarks, DelveAgent improves accuracy by up to 7.5 percentage points while reducing inference costs to approximately one-third of the strongest baseline. These results establish the significance of PhySciBench as a critical benchmark for evaluating AI systems in the physical sciences and demonstrate that architectural specialization can effectively enhance the reliability of autonomous scientific research.
