Search papers, labs, and topics across Lattice.
This paper introduces SceneActBench, a novel benchmark designed to evaluate vision-language model (VLM) agents on their ability to perform actions in multi-object 3D scenes. By addressing the limitations of existing benchmarks that primarily focus on textual responses or single-object operations, SceneActBench allows for a comprehensive assessment of agent performance across five diverse tasks using a unified agent-environment loop. The results reveal a significant variability in performance among eleven VLM configurations, with overall scores ranging from 38.6 to 50.2, highlighting the challenges agents face in executing complex actions in 3D environments.
Agents struggle to act effectively in 3D scenes, with none of the eleven evaluated VLMs achieving consistent performance across diverse tasks.
Vision-language model (VLM) agents increasingly use tools to act on 3D scenes rather than only describe them. Existing 3D benchmarks score textual responses or single-object operations, leaving agent action on complete multi-object 3D scenes under evaluated. We present SceneActBench, a benchmark for visually conditioned action across five 3D tasks under a unified agent-environment loop. Given PNG images or sampled video frames and, where applicable, supplied 3D assets, an agent acts on a 3D environment. We evaluate each final output against hidden ground truth with task-specific geometric metrics. SceneActBench comprises five tasks built from 210 source instances, yielding 520 task cases including paired input conditions. Every task runs through one fixed agent loop to keep the comparison fair. Across eleven proprietary VLM configurations, Overall scores span 38.6-50.2, and none performs consistently well across tasks. We further analyse where and how failures manifest.
