Search papers, labs, and topics across Lattice.
This paper introduces SciFigBench, a benchmark designed to evaluate vision-language models (VLMs) on their ability to understand scientific figures while accounting for behavioral reliability under uncertainty. By employing the Admittance-Resistance-Inductance (A-R-I) framework, the authors assess how well models acknowledge insufficient evidence and resist misleading context, revealing significant behavioral discrepancies among leading models. Notably, while GPT-5.2 excels in description quality and reasoning accuracy, it frequently hallucinates content, whereas Gemini 3.1 Pro demonstrates greater reliability by admitting uncertainty in a majority of cases, highlighting the importance of behavioral reliability in scientific applications.
High reasoning accuracy in VLMs doesn't equate to reliability, as shown by GPT-5.2's 96% hallucination rate despite top performance metrics.
Existing vision-language model (VLM) benchmarks emphasize perception and reasoning accuracy (how well VLMs describe and reason about what they see in an image), with limited attention to behavioral reliability under uncertainty (how they behave when visual evidence is missing or misleading). We introduce SciFigBench, a diagnostic VLM benchmark for scientific figure understanding that jointly evaluates perception, reasoning, and behavioral reliability under uncertainty. It contains 250 figures with high-quality human annotations across three evaluation aspects, totaling 600+ hours of annotation effort. We further extend these figures via image transformations, reasoning questions, resistance probes, caption-bias probes, and confirmed selective-blur targets, producing over 34,000 evaluation setups for stress testing. We further propose the Admittance-Resistance-Inductance (A-R-I) framework to evaluate whether models acknowledge insufficient evidence, resist misleading context, and infer cautiously from partial information. Our results reveal substantial behavioral differences among models. GPT-5.2 achieves the highest description quality (MQM 91.6) with strong reasoning accuracy (78.4%), yet hallucinates unreadable content in 96% of cases, whereas Gemini 3.1 Pro, a comparably capable model (MQM 90.2, reasoning 81.0%), admits uncertainty in 71% of such cases and achieves the strongest resistance score (0.91). These findings show that high perception and reasoning accuracy alone do not guarantee behavioral reliability, a dimension critical for deployment in scientific workflows.
