Search papers, labs, and topics across Lattice.
This paper introduces GaussFusion, a multimodal pre-training framework that enhances 3D Gaussian representations by integrating image and text supervision through cross-modal semantic alignment. The method addresses the limitations of existing masked Gaussian reconstruction techniques, which primarily focus on local structures, by employing Gaussian Salience-guided Multi-scale Hole Masking (GSHM) to create spatially continuous masked regions that capture both local and global features. Experimental results show that GaussFusion significantly improves the transferability of Gaussian representations, outperforming previous methods on benchmark datasets such as ModelNet40 and ScanObjectNN.
GaussFusion not only enhances 3D Gaussian representations but also achieves a remarkable improvement in transferability, outperforming existing methods by notable margins.
3D Gaussian Splatting provides an explicit representation that jointly models geometry and appearance, serving as a scalable foundation for 3D representation learning. Existing pre-training methods for Gaussian representations, such as masked Gaussian reconstruction, primarily capture local structures but offer limited semantic supervision. In this paper, we propose GaussFusion, a multimodal pre-training framework for 3D Gaussian representations. GaussFusion integrates image and text supervision into masked Gaussian modeling through cross-modal semantic alignment, enabling the Gaussian encoder to learn both visual and language-level semantic information during pre-training. To better adapt masked modeling to the non-uniform distribution of Gaussian primitives, we further propose Gaussian Salience-guided Multi-scale Hole Masking (GSHM). GSHM constructs spatially continuous masked regions based on Gaussian salience. By applying hole masks at multiple scales, GSHM encourages the encoder to capture both fine-grained local patterns and broader structural dependencies. Extensive experiments on downstream tasks demonstrate that GaussFusion improves the transferability of Gaussian representations. Notably, GaussFusion outperforms Gaussian-MAE on ModelNet40 and ScanObjectNN (PB-T50-RS) by 0.61\% and 3.85\%, respectively.
