Search papers, labs, and topics across Lattice.
This paper introduces Manipulation-Grounded Visual Question Answering (MG-VQA) as a new framework for evaluating vision-language models (VLMs) in dynamic, cluttered environments where physical manipulation of objects is necessary to answer questions. The authors develop PROBE, a simulator and benchmark suite that includes 150 tasks requiring VLMs to perceive and manipulate objects before responding, revealing that agentic tool-based methods significantly outperform perception-only approaches. Additionally, they present PROBE-Agent, a finetuning method that enhances smaller models' performance through knowledge distillation from larger models, achieving notable improvements in both simulated and real-world settings.
Agentic manipulation in VLMs leads to a consistent 8% performance boost over perception-only methods in cluttered environments, demonstrating the critical role of physical interaction in visual question answering.
Vision-language Models (VLMs) excel at 2D grounding, spatial reasoning and agentic tool-based planning in static scenes. However, consider asking a home robot "Is my medication still in the cabinet?" The answer may be physically hidden behind a row of containers that must first be moved aside. Answering such questions in real-world cluttered environments requires reasoning in dynamic scenes: distractors must be manipulated to reveal occluded objects, and each action changes the scene the model must reason over. We formalize this setting as Manipulation-Grounded Visual Question Answering (MG-VQA) and introduce PROBE, a framework for benchmarking and finetuning VLM agents on such tasks. We first develop PROBE-Sim, a high-fidelity tabletop simulator with everyday objects and a robot manipulator equipped with grasping and pushing tools. PROBE-Sim is used to create PROBE-Bench: an evaluation suite of 150 tasks across 6 question types on cluttered tabletop scenes, where a VLM perceives, picks up or pushes objects before answering. We observe consistent trend across all frontier VLMs: agentic tool-based methods outperform their perception-only baselines (8.0% on average) across all task types. We further design PROBE-Agent, a finetuning recipe to distill successful trajectories from a powerful teacher foundation model to a smaller open-weight model using a mixed data recipe that encourages manipulation-efficient question answering. PROBE Agent finetuned models outperform their off-the-shelf agent baseline (11.5% on average) and demonstrate positive transfer to unseen objects and a held-out task. We validate sim-to-real transfer by deploying PROBE-Agent finetuned policies in real-world tabletop environments.
