Search papers, labs, and topics across Lattice.
This paper introduces InSituMeasure, a benchmark designed to evaluate the performance of Multimodal Large Language Models (MLLMs) in the context of situated measurement grounding within industrial environments. The dataset comprises 2,922 real industrial scenes annotated with gauge attributes and noise tags, highlighting the challenges faced by MLLMs in achieving reliable continuous-valued measurements. Results show that even the best-performing model achieves only 25.7% joint value-unit accuracy, indicating a significant gap in MLLMs' capabilities when applied to realistic, noisy industrial settings compared to their performance on general multimodal tasks.
MLLMs struggle with real-world industrial measurements, achieving only 25.7% accuracy in a new benchmark that simulates authentic operational challenges.
For trained operators, gauge reading requires little specialized knowledge, low cognitive effort, and high repeatability. Yet Multimodal Large Language Models (MLLMs) remain unreliable in continuous-valued measurement despite strong results on general multimodal benchmarks. Existing benchmarks expose this weakness but isolate measurement from realistic, knowledge-grounded settings, with limited situated context, specialized instruments, real-world noise, and matched diagnostic annotations, reducing realism and constraining root-cause analysis. We introduce InSituMeasure to evaluate situated measurement grounding. It contains 2,922 real industrial monitoring scenes across eight functional categories of professional engineering instruments, with dense gauge-attribute annotations and noise tags for failure diagnosis. We define metrics for numerical accuracy under predefined tolerances and unit consistency, rejection of fake or unanswerable tasks, and alignment between model failures and annotated error factors. Across 24 state-of-the-art MLLMs, the best model reaches only 25.7\% joint value-unit accuracy and 51.8\% confidence-diagnosis F1, revealing a substantial gap between general multimodal competence and reliable situated measurement. Further analysis identifies failures from text-induced shortcuts, overconfident responses, and authentic industrial noise, including mixed disturbances, viewpoint deviation, occlusion, and environmental interference.
