Search papers, labs, and topics across Lattice.
This paper introduces GaussianFusion, a novel framework that utilizes a 3D Gaussian representation for multi-modal fusion perception, moving away from the traditional bird's-eye view (BEV) approach. By employing a forward-projection-based multi-modal Gaussian initialization module and a shared cross-modal Gaussian encoder, the model effectively preserves fine details and enhances feature alignment across modalities. Experimental results on the nuScenes dataset show that GaussianFusion outperforms existing methods, achieving a 2.6 NDS improvement over BEVFusion and a significant speedup with fewer Gaussians compared to GaussFormer.
GaussianFusion achieves superior 3D perception by leveraging a continuous 3D Gaussian space, outperforming traditional methods while using fewer resources.
The bird's-eye view (BEV) representation enables multi-sensor features to be fused within a unified space, serving as the primary approach for achieving comprehensive 3D perception. However, the discrete grid representation of BEV leads to significant detail loss and limits feature alignment and cross-modal information interaction in multimodal fusion perception. In this work, we break from the conventional BEV paradigm and propose a new universal framework for multi-modal fusion based on 3D Gaussian representation. This approach naturally unifies multi-modal features within a shared and continuous 3D Gaussian space, effectively preserving edge and fine texture details. To achieve this, we design a novel forward-projection-based multi-modal Gaussian initialization module and a shared cross-modal Gaussian encoder that iteratively updates Gaussian properties based on an attention mechanism. GaussianFusion is inherently a task-agnostic model, with its unified Gaussian representation naturally supporting various 3D perception tasks. Extensive experiments demonstrate the generality and robustness of GaussianFusion. On the nuScenes dataset, it outperforms the 3D object detection baseline BEVFusion by 2.6 NDS. Its variant surpasses GaussFormer on 3D semantic occupancy with 1.55 mIoU improvement while using only 30% of the Gaussians and achieving a 450% speedup.
