Search papers, labs, and topics across Lattice.
This paper introduces CAIRN, a topology-aware 3D-LLM designed for understanding complex multi-room 3D scenes, addressing the limitations of existing models that only handle single-room environments. By integrating transformer attention with scene hierarchy and enhancing object tokens with room-local relational context through a graph neural network, CAIRN achieves superior performance on a newly established benchmark, CAIRN-MR, which includes diverse tasks from grounding to cross-room reasoning. Experimental results demonstrate that CAIRN significantly outperforms previous models in multi-room scenarios while maintaining competitiveness in single-room tasks.
CAIRN redefines 3D scene understanding by seamlessly integrating room-level topology with object-level relations, achieving unprecedented performance in multi-room environments.
Existing 3D scene-grounded Large Language Models (3D-LLMs) focus on answering questions grounded in simplified single-room 3D scenes, lacking the ability to reason over real-world household environments containing multiple interconnected rooms and diverse object categories. We introduce CAIRN, a topology-aware 3D-LLM for multi-room 3D scene understanding. CAIRN aligns transformer attention with scene hierarchy, giving the model explicit awareness of object-level relations and room-level connectivity. It enriches object tokens with room-local relational context via a graph neural network, introduces learned room tokens for room-level abstraction, and applies a hierarchical attention mask with geometric bias to route information according to scene topology. CAIRN is developed on CAIRN-MR, a benchmark we introduce on HM3D for multi-room 3D scene understanding, covering grounding, captioning, and four question-answering tasks that progressively evaluate from intra-room perception to cross-room reasoning. Experiments show that CAIRN outperforms prior 3D-LLMs by a large margin across all CAIRN-MR tasks while remaining competitive on five single-room benchmarks.
Microsoft Research