Search papers, labs, and topics across Lattice.
The paper introduces SPM-Bench, a novel multimodal benchmark designed to evaluate LLMs' proficiency in scanning probe microscopy (SPM) at a PhD level. They present a fully automated data synthesis pipeline leveraging Anchor-Gated Sieve (AGS) technology to extract high-value image-text pairs from scientific literature and a hybrid cloud-local architecture for efficient data processing. The authors introduce the Strict Imperfection Penalty F1 (SIP-F1) score to evaluate LLM performance and quantify model "personalities," revealing reasoning boundaries in complex physical scenarios.
LLMs still struggle with PhD-level scanning probe microscopy tasks, but SPM-Bench offers a new automated pipeline to generate challenging scientific benchmarks and quantify model "personalities" like "Conservative" or "Gambler."
As LLMs achieved breakthroughs in general reasoning, their proficiency in specialized scientific domains reveals pronounced gaps in existing benchmarks due to data contamination, insufficient complexity, and prohibitive human labor costs. Here we present SPM-Bench, an original, PhD-level multimodal benchmark specifically designed for scanning probe microscopy (SPM). We propose a fully automated data synthesis pipeline that ensures both high authority and low-cost. By employing Anchor-Gated Sieve (AGS) technology, we efficiently extract high-value image-text pairs from arXiv and journal papers published between 2023 and 2025. Through a hybrid cloud-local architecture where VLMs return only spatial coordinates"llbox"for local high-fidelity cropping, our pipeline achieves extreme token savings while maintaining high dataset purity. To accurately and objectively evaluate the performance of the LLMs, we introduce the Strict Imperfection Penalty F1 (SIP-F1) score. This metric not only establishes a rigorous capability hierarchy but also, for the first time, quantifies model"personalities"(Conservative, Aggressive, Gambler, or Wise). By correlating these results with model-reported confidence and perceived difficulty, we expose the true reasoning boundaries of current AI in complex physical scenarios. These insights establish SPM-Bench as a generalizable paradigm for automated scientific data synthesis.
DAMO