Search papers, labs, and topics across Lattice.
The authors construct LogiScope-VQA, an industrial safety benchmark comprising 2,476 images, 2,918 real-world logistics videos, and 10,274 human-verified QA pairs structured across 39 perception, domain-knowledge, and hazard-reasoning subtasks. Evaluating vision-language models on industrial hazard identification is critical for autonomous physical deployment, where visual edge cases directly impact operational safety. Benchmark results show that frontier proprietary models severely underperform human experts when jointly integrating visual perception with domain-specific risk reasoning, while exhibiting pervasive security biases across varying inference compute budgets.
Industrial warehouse safety exposes a massive capability gap in frontier LMMs, where even top proprietary models consistently fail to jointly ground visual hazards and reason over operational risks.
Large Multimodal Models (LMMs) large-scale deployment in industrial warehouse settings specifically necessitates that models exhibit human-expert-level hazard-oriented perception, understanding, and reasoning capabilities. However, the scarcity of real industrial data, tightly coupled to commercial terms, significantly hampers further advancement. To bridge this gap, we curate LogiScope-VQA to investigate the practical applicability of mainstream LMMs in real-world logistics operations. LogiScope-VQA comprises 2,476 images and 2,918 videos primarily sourced from real-world logistics parks, along with 10,274 VQAs meticulously curated and validated by human annotators. Grounded in 18 core objects and 20 risk types, we devise 39 subtasks aligned with three principal themes: industrial element perception, warehouse knowledge understanding, and potential risk reasoning. Furthermore, we incorporate dynamic thinking-budget configurations and dual-dimensional risk bias analyses to elucidate the properties of LMMs. Extensive experiments unveil that even powerful proprietary models, including GPT-5.5, Gemini-3.1-Pro, and Claude-Opus-4.7, exhibit a significant gap relative to human performance. The unique challenge of jointly integrating perception, understanding, and reasoning for hazard identification poses substantial headroom for further improvement on LogiScope-VQA. We additionally reveal the pervasive security bias issue that impedes LLMs'practical deployment in real-world settings. The industrial dataset is publicly available under the CC BY-NC-SA 4.0 license.
