Search papers, labs, and topics across Lattice.
This paper introduces a collaborative multi-agent framework for zero-shot 3D understanding, leveraging a Planning Agent for viewpoint optimization and a Perception Agent for creating a structured cognitive map of the scene. By iteratively refining their collaboration, the agents enhance the observation of 3D environments, overcoming the limitations of traditional video-based methods. The approach achieves state-of-the-art performance across six benchmarks, significantly improving accuracy and dialogue metrics in 3D understanding tasks.
A novel multi-agent framework enhances zero-shot 3D understanding by iteratively optimizing viewpoints and integrating fragmented observations, leading to significant performance gains.
Recent advancements have explored agentic zero-shot 3D understanding by reformulating it as video keyframe understanding with Multimodal Large Language Models (MLLMs). However, existing methods face an intrinsic bottleneck due to the finite observation perspectives inherent in videos and the implicit perception of 3D scenes. In this paper, we propose a collaborative multi-agent framework that assigns a Planning Agent to handle high-level viewpoint planning and supplement novel perspectives, and a Perception Agent to explicitly summarize the 3D scene into a structured holistic cognitive map. Specifically, Planning Agent first analyzes this cognitive map to determine query-relevant viewpoints and supplements missing critical perspectives to ensure comprehensive observation. Subsequently, Perception Agent documents object-level attributes from these views by assigning consistent instance identifiers across viewpoints, thereby integrating fragmented observations into the holistic cognitive map. In parallel, it provides feedback to filter out mismatched candidate objects and guide subsequent viewpoint planning. Through this closed-loop iterative process, two agents collaboratively figure out candidates until Perception Agent determines that sufficient information has been captured to complete the task. Extensive experiments demonstrate that our method achieves state-of-the-art performance on 6 benchmarks, with improvements of 11.1\% Acc@0.5 on ScanRefer, 14.6 BLEU-1 on 3D-assisted dialog, and 2.1 EM on SQA3D.
