Search papers, labs, and topics across Lattice.
This paper introduces SteelBench, a novel benchmark designed to evaluate vision-language models (VLMs) in real-world industrial settings, addressing the limitations of existing benchmarks that focus on consumer videos or simulated environments. By analyzing 1,345 densely annotated clips from actual industrial CCTV footage, the study highlights significant performance gaps, revealing that the best VLM only achieves 42.6% action accuracy compared to an 84.6% human benchmark. Additionally, the findings underscore the impact of annotation provenance, demonstrating that unaudited labels can artificially inflate model accuracy by up to 17 percentage points, emphasizing the need for rigorous evaluation protocols in industrial applications.
Unaudited labels can inflate VLM accuracy by up to 17 percentage points, revealing critical flaws in current evaluation practices for industrial AI systems.
Existing video benchmarks evaluate action recognition on consumer videos, egocentric recordings, or simulated industrial environments. They do not test vision-language models under the visual and procedural conditions of real industrial CCTV, where workers appear as distant figures amid dust, steam, low light, glare, occlusion, and overlapping activities. We introduce STEELBENCH, a diagnostic benchmark for industrial surveillance that jointly evaluates per-worker activity recognition, safety-rule reasoning, and annotation provenance. SteelBench contains 1,345 densely annotated clips, curated from 149 hours of operational plant footage and 10,024 candidate clips using temporal deduplication, class balancing, and visibility-aware stratified sampling. Each clip includes dense per-worker action labels, PPE attributes, spatial context, and safety-rule annotations. Because model-assisted annotation can shape the labels later used for model evaluation, SteelBench includes a provenance-aware audit protocol. The protocol measures label influence, evaluates sensitivity to ground-truth provenance, and reports a human reference from expert-reviewed labels. Applying this audit, we find that unaudited VLM-sourced ground truth can inflate same-family model accuracy by up to 17 percentage points. Across nine VLMs from four architectural families, the best model reaches only 42.6% action accuracy, compared with an 84.6% human benchmark. Performance also fragments across recognition, robustness, calibration, and safety reasoning. Even when models predict the correct action, 37-58% of cases still yield incorrect safety judgments, and no model passes more than 2 of 5 diagnostic checks. The dataset is publicly available on Hugging Face.
