Search papers, labs, and topics across Lattice.
EarthVerse introduces a comprehensive benchmark for evaluating scientific agents tasked with analyzing dynamic Earth systems and natural hazards through 405 reproducible tasks based on 199 documented events. The study highlights significant shortcomings in current agents, revealing that while the best systems achieve a mean answer-unit accuracy of 84.65%, only 34.81% meet strict criteria for maintaining a coherent chain of evidence and reasoning. This benchmarking framework not only assesses individual task performance but also emphasizes the importance of preserving provenance and transparency in scientific inquiry.
Current scientific agents struggle to maintain a coherent narrative across evidence and calculations, with only 34.81% achieving strict accuracy in complex tasks.
Earth-system analysis reconstructs changing physical processes from observations that differ in source, scale, timing, and modality. Natural hazards make this work consequential because incomplete evidence can change estimates of severity, exposure, and mechanism. We introduce EarthVerse, a benchmark that evaluates scientific agents through package-scoped investigations. Its 405 reproducible tasks are grounded in 199 documented events and 19 hazard families. Agents inspect heterogeneous event packages, choose compatible evidence, execute transparent calculations, reconcile source differences, and preserve provenance in the final answer. We provide executable ground truth that decomposes each task into fine-grained answer units, together with task-specific rubrics that assess the supporting research process while allowing multiple valid paths. We evaluate 25 model and agent systems under a controlled tool-using protocol, then use controlled studies to locate failures in evidence access, tool selection, memory, reasoning, interaction, and scientific execution. Across systems, the best mean answer-unit accuracy is 84.65%, while the highest Strict@95 is only 34.81%. The gap shows that current agents often complete individual steps without maintaining a consistent chain across evidence, scales, units, calculations, and physical interpretation. EarthVerse provides a reproducible basis for measuring end-to-end scientific reliability in dynamic Earth systems.
Tsinghua AI