Search papers, labs, and topics across Lattice.
This paper introduces VANTAGE-Bench, a benchmark designed to evaluate the performance of Vision-Language Models (VLMs) in the context of Infrastructure AI, which is often overlooked in favor of action-oriented Embodied AI. By examining three operational domains鈥擫ogistics, Transportation, and Smart Spaces鈥擵ANTAGE-Bench assesses models across a variety of tasks, revealing significant performance gaps in event verification, referring expressions, and temporal localization. The findings indicate that while some models perform well in specific areas, they consistently underperform in temporal tasks, highlighting a critical gap in current VLM capabilities for fixed-camera infrastructure applications.
Infrastructure AI reveals a hidden performance gap in VLMs, with temporal tasks lagging significantly behind consumer-centric benchmarks.
As Vision-Language Models (VLMs) advance toward physical deployment, the focus has remained on action-oriented Embodied AI evaluated on subject-centric consumer video. This overlooks a pervasive class of Physical AI: Infrastructure AI, which relies on fixed cameras for open-loop insights like safety monitoring and operational logging. We introduce VANTAGE-Bench, a benchmark measuring this"Infrastructure AI Gap."It spans three operational domains (Logistics, Transportation, and Smart Spaces), unifies image and video evaluation across semantic, spatial, temporal, and spatio-temporal capabilities, and moves beyond multiple-choice to eight task formulations including dense captioning and spatio-temporal grounding. It adds a single-pass trajectory protocol for Single Object Tracking and, to our knowledge, the first such evaluation on fixed-camera infrastructure video, scored against specialist trackers. Annotation spans three regimes over 3,346 media assets: 3,342 video-task annotations, 4,281 image-grounding annotations, and 27,404 detection boxes. Evaluating 17 models zero-shot, we find the shortfall relative to consumer-centric benchmarks is concentrated, not general. Event verification, referring expressions, and temporal localization fall roughly 9 to 24 points at every model scale, while video question answering stays within 5.3 points of VideoMME and 2D spatial pointing shows no shortfall against BLINK. The temporal pillar is weakest in absolute terms: no system exceeds 55.7 mIoU on temporal localization or 37.3 SODA_c on dense video captioning. On tracking, frontier models come within roughly 5 points of specialist trackers over short horizons but separate as the horizon extends. Open-weight models lead 2D object localization outright, so neither scale nor proprietary access explains the pattern. Data, evaluation harness, and leaderboard: https://vantage-bench.org/
