Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
The authors construct JRDB-AVR, an embodied benchmark on real-world robotics data that requires agents to actively retrieve bounded spatio-temporal observations (timestamps and viewing angles) to answer visual reasoning questions. This addresses a critical flaw in passive benchmarks, where models frequently arrive at correct answers via spurious correlations without ever observing the underlying evidence. Evaluating modern VLMs alongside an observation-grounded graph-based reference agent reveals a severe discrepancy between answer accuracy and evidence accuracy, underscoring the prevalence of ungrounded reasoning in embodied systems.
Embodied VLMs routinely succeed for the wrong reasons, answering visual queries correctly despite completely missing the critical timestamps and viewpoints required to ground their answers.
In complex embodied visual reasoning scenarios, an agent often has only a limited field of view, and the evidence needed to answer a question may be distributed across time, viewpoint, and interacting objects. A model may therefore give a plausible answer without ever observing the relevant object, time, or view that supports it. Current visual reasoning benchmarks largely evaluate passive observations and final answers, overlooking settings that require active reasoning and evidence acquisition. We introduce JRDB-AVR, a benchmark derived from existing real-world JRDB robotics data through a structured question-generation engine that turns this gap into an explicit evaluation: an embodied agentic system receives a visual reasoning question, requests bounded observations by timestamp and viewing angle, and is evaluated on both the final answer and the grounded visual evidence supporting it. The benchmark contains diverse questions over multiple real-world environments involving temporal search, viewpoint selection, and human-oriented compositional reasoning. We also introduce JRDB-AVR-Agent, a reference active reasoning agentic method that maintains an explicit observation-grounded graph-based world model and answers through solving. Experiments reveal a substantial gap between answer accuracy and evidence accuracy in current baselines, showing that current VLMs can produce unsupported correct answers and that active evidence-aware evaluation is necessary for embodied visual reasoning. Code and benchmark are available at https://github.com/ControlNet/JRDB-AVR.