Search papers, labs, and topics across Lattice.
This paper introduces SCOPE, a benchmark for evaluating the ability of large language models (LLMs) to conduct high-quality experimental design across 19 research domains. The study reveals that most LLMs struggle with both high-level planning completeness and low-level configuration accuracy, with a notable performance bottleneck in the latter. To address these limitations, the authors propose OptED, an agentic workflow that optimizes LLM-based experimental design through stage isolation, tool augmentation, and rule-based constraints, significantly improving the quality of experimental planning.
Most LLMs fail to design high-quality experiments, revealing a critical gap in AI's role in scientific research.
AI for Research (AI4Research) leverages AI to automate and improve scientific workflows. While experimental design is a critical stage of the research process, prior work has focused primarily on code implementation and execution, overlooking the importance of this stage, and no benchmark exists to evaluate AI's ability to conduct systematic experiment design. To bridge this gap, we propose SCOPE, a Scientific COmprehensive Planning Evaluation Benchmark constructed from 300 high-quality latest papers across 19 research domains from top-tier venues (e.g., ICML, NeurIPS, and ICLR),evaluating LLMs on two dimensions: High-Level planning completeness (main, ablation, and analysis experiments) and Low-Level configuration accuracy and rationality (datasets, baselines, and metrics). Benchmarking reveals three findings: (1) most LLMs cannot directly design high-quality experiments; (2) all LLMs exhibit a performance bottleneck in low-level configuration; and (3) search mode does not improve design quality. Furthermore, to address these challenges, we propose OptED, a novel agentic workflow to optimize LLM-based experimental design, that enhances LLM-based experimental planning through stage isolation, tool augmentation, and rule-based constraints, effectively alleviating the configuration bottleneck.
