Search papers, labs, and topics across Lattice.
This paper introduces MM-TRELLIS, a novel multi-modal approach for generating realistic 3D vehicle models by leveraging both multi-view images and LiDAR point clouds from autonomous driving datasets. The method addresses the limitations of existing neural rendering techniques by employing a denoising process that aligns LiDAR guidance with model priors and enforces geometric consistency, resulting in high-fidelity mesh outputs. Experimental results on the Waymo dataset show that MM-TRELLIS significantly outperforms prior methods in generating accurate 3D vehicle representations.
MM-TRELLIS achieves unprecedented fidelity in 3D vehicle generation by seamlessly integrating multi-view images and LiDAR data, outperforming existing methods in real-world scenarios.
Recovering realistic 3D vehicle models from autonomous driving scenes is crucial for synthesizing training data and building simulation environment. However, most existing vehicle generation methods fail to fully exploit multimodal sensors i.e. multi-view images and LiDAR point clouds) and rely on neural rendering based reconstruction, leading to low-quality mesh. Recently, native 3D generative models have made significant progress, yet they are not built for arbitrary multi-view inputs and often struggle with in-the-wild driving images. In this work, we present MM-TRELLIS, a multi-modal version of TRELLIS for in-the-wild 3D vehicle generation that integrates LiDAR and image sensors from autonomous driving datasets into native 3D generative models. Specifically, multi-view images are cycled as conditioning inputs, while LiDAR point clouds provide test-time guidance to ensure geometric accuracy and cross-view consistency. During denoising, we first align the guidance point cloud with the model priors, then enforce consistency between the generated geometry and the guidance point cloud. Finally, we introduce a voxel filtering strategy based on the opacity of 3D Gaussian Splatting to suppress floaters and produce clean meshes. Comprehensive experiments on Waymo dataset demonstrate our method outperforms existing methods in high-fidelity 3D vehicle generation. Code is available at https://github.com/HongliXiao/MM-TRELLIS.
