Search papers, labs, and topics across Lattice.
This paper introduces {\Psi}-Map, a novel framework for open-vocabulary panoptic reconstruction that combines geometric reinforcement with end-to-end panoptic learning and efficient rendering. It uses LiDAR data to construct plane-constrained multimodal Gaussian Mixture Models (GMMs) with 2D Gaussian surfels for high-precision surface alignment and continuous geometric supervision. The system lifts 2D mask features directly into 3D space using a query-guided end-to-end learning architecture with local cross-attention, achieving globally consistent panoptic understanding at over 40 FPS.
Achieve real-time, high-quality panoptic 3D reconstruction by directly lifting 2D features into 3D space, bypassing traditional multi-stage pipelines.
Open-vocabulary panoptic reconstruction is essential for advanced robotics perception and simulation. However, existing methods based on 3D Gaussian Splatting (3DGS) often struggle to simultaneously achieve geometric accuracy, coherent panoptic understanding, and real-time inference frequency in large-scale scenes. In this paper, we propose a comprehensive framework that integrates geometric reinforcement, end-to-end panoptic learning, and efficient rendering. First, to ensure physical realism in large-scale environments, we leverage LiDAR data to construct plane-constrained multimodal Gaussian Mixture Models (GMMs) and employ 2D Gaussian surfels as the map representation, enabling high-precision surface alignment and continuous geometric supervision. Building upon this, to overcome the error accumulation and cumbersome cross-frame association inherent in traditional multi-stage panoptic segmentation pipelines, we design a query-guided end-to-end learning architecture. By utilizing a local cross-attention mechanism within the view frustum, the system lifts 2D mask features directly into 3D space, achieving globally consistent panoptic understanding. Finally, addressing the computational bottlenecks caused by high-dimensional semantic features, we introduce Precise Tile Intersection and a Top-K Hard Selection strategy to optimize the rendering pipeline. Experimental results demonstrate that our system achieves superior geometric and panoptic reconstruction quality in large-scale scenes while maintaining an inference rate exceeding 40 FPS, meeting the real-time requirements of robotic control loops.
