Search papers, labs, and topics across Lattice.
This paper introduces a novel memory tree guided key frame selection method, MemTree3D, designed to enhance the efficiency of 3D question answering in embodied scenarios by leveraging a compact 3D scene representation. By allowing real-time online construction and multi-level scene information capture, the approach significantly reduces the computational burden of visual search methods, which traditionally require extensive frame retrieval for each query. The results demonstrate a notable improvement in performance, with LLM-Match scores increasing by 17.4% for GPT-4o and 5.8% for LLaVA-OneVision-7B, outperforming existing methods.
MemTree3D slashes the computational cost of 3D question answering, boosting performance while avoiding the inefficiencies of traditional visual search methods.
Answering questions accurately and efficiently in embodied scenarios presents significant challenges due to limited computational and memory resources for Vision Language Model (VLM) inference. Existing methods adopt visual search key frame retrieval method to select critical question-related key frames for VLM input. However, visual search methods are inefficient because they require visual search among thousands of video frames for each individual user query. In this work, we propose a memory tree guided key frame selection paradigm for efficient 3D question answering in embodied scenarios. Our method leverages a compact and reusable 3D scene representation, termed MemTree3D, which supports real-time online construction leveraging camera 6-DoF poses. MemTree3D captures multi-level 3D scene information, enabling a Large Language Model to efficiently query and retrieve question-relevant key frames through our scoring-based frame selection without reprocessing the entire video stream. On OpenEQA, our method improves the LLM-Match of GPT-4o by 17.4%, LLaVA-OneVision-7B by 5.8%, outperforms existing visual search methods. Our code is available at https://github.com/hsiangwei0903/MemTree3D
Amazon Science