Search papers, labs, and topics across Lattice.
The paper introduces X-GS, an open framework unifying various 3D Gaussian Splatting (3DGS) techniques for real-time online SLAM and bridging the gap to downstream multimodal models. X-GS employs X-GS-Perceiver, an efficient pipeline that co-optimizes geometry and poses from unposed RGB/RGB-D video streams while distilling semantic features from vision foundation models into the 3D Gaussians. The framework achieves real-time performance through a novel online Vector Quantization module, GPU-accelerated grid-sampling, and parallelized pipeline design, enabling downstream tasks like object detection and zero-shot captioning via the X-GS-Thinker component.
Unlock real-time semantic SLAM and multimodal interaction with 3D Gaussian Splatting using X-GS, a unified and extensible open framework.
3D Gaussian Splatting (3DGS) has emerged as a powerful technique for novel view synthesis, subsequently extending into numerous spatial AI applications. However, most existing 3DGS methods are isolated, focusing on specific domains such as online SLAM, semantic enrichment, or 3DGS for unposed images. In this paper, we introduce X-GS, an extensible open framework that unifies a broad range of techniques to enable real-time 3DGS-based online SLAM enriched with semantics, bridging the gap to downstream multimodal models. At the core of X-GS is a highly efficient pipeline called X-GS-Perceiver, capable of taking unposed RGB (or optionally RGB-D) video streams as input to co-optimize geometry and poses, and distill high-dimensional semantic features from vision foundation models into the 3D Gaussians. We achieve real-time performance through a novel online Vector Quantization (VQ) module, a GPU-accelerated grid-sampling scheme, and a highly parallelized pipeline design. The semantic 3D Gaussians can then be utilized by vision-language models within the X-GS-Thinker component, enabling downstream tasks such as object detection, zero-shot caption generation, and potentially embodied tasks. Experimental results on real-world datasets showcase the efficacy, efficiency, and newly unlocked multimodal capabilities of the X-GS framework.
