Search papers, labs, and topics across Lattice.
This paper introduces DR-MV3D, a novel map-grounded learning framework designed to enhance Multi-view 3D Visual Question Answering (MV3D-VQA) by providing dense, verifiable rewards that improve the integration of partial observations into coherent 3D scene representations. By decomposing the MV3D-VQA task into global map construction, question-conditioned view-trajectory planning, and egocentric grounding, the authors implement two innovative reward mechanisms: a global consistency reward and a local trajectory reward, which facilitate effective learning without manual annotations. Experimental results on multiple benchmarks demonstrate that DR-MV3D significantly outperforms existing multi-image baselines, highlighting the advantages of dense supervision in complex reasoning tasks.
Dense rewards can transform how we approach multi-view reasoning, leading to substantial performance gains in 3D visual question answering.
Multi-view 3D Visual Question Answering (MV3D-VQA) requires integrating partial observations into a coherent 3D scene representation and selecting informative viewpoints for multi-step spatial reasoning. However, current multimodal LLMs are typically trained with sparse, answer-level supervision, which often yields inconsistent cross-view reasoning and brittle view selection. We present DR-MV3D (Dense Reward for MV3D-VQA), a map-grounded learning framework that provides dense, verifiable rewards to supervise the reasoning process. Our approach decomposes MV3D-VQA into (i) allocentric global map construction, (ii) question-conditioned view-trajectory planning, and (iii) egocentric grounding for answer prediction. To make intermediate steps learnable without manual annotations, we introduce two rewards: a global consistency reward that aligns the predicted map with geometry-consistent pseudo targets from frozen 3D vision foundation models (e.g., VGGT + SAM3), and a local trajectory reward that supervises ordered viewpoint selection. We optimize the full pipeline with trajectory-level policy optimization (GRPO). Experiments on MindCube, VSI-Bench, and BLINK (MV) show that DR-MV3D consistently improves over strong multi-image baselines, supporting the effectiveness of process-level dense supervision for multi-view 3D reasoning.
