Search papers, labs, and topics across Lattice.
This paper introduces T2LDM++, a novel Text-to-LiDAR diffusion model that addresses the challenges of generating realistic LiDAR scenes from textual descriptions by employing a Self-Conditioned Representation Guidance (SCRG) mechanism. By utilizing a Guidance Network to provide reconstruction-based supervision, the model significantly enhances the geometry-aware representations, resulting in improved denoising and scene fidelity while maintaining computational efficiency. The authors also create two extensive Text-LiDAR benchmarks and demonstrate that T2LDM++ can effectively generate high-quality LiDAR scenes under various conditions, showcasing its robustness and versatility.
T2LDM++ generates realistic LiDAR scenes with rich geometric details, overcoming the limitations of existing models that struggle with insufficient training data and controllability.
Recent progress in Text-to-Image generation benefits from large-scale Text-Image pairs. However, the scarcity of Text-LiDAR pairs often causes over-smoothed scenes and limited controllability. In this paper, we rethink the limitations of Text-LiDAR generation task, focusing on alleviating insufficient training priors and constructing controllable Text-LiDAR data. We propose a \textbf{T}ext-\textbf{to}-\textbf{L}iDAR \textbf{D}iffusion \textbf{M}odel for LiDAR scene generation, T2LDM++, with a Self-Conditioned Representation Guidance (SCRG). Specifically, to alleviate object over-smoothing, SCRG employs a Guidance Network (GN) to provide reconstruction-based soft supervision to the Denoising Network (DN). This enables DN to learn geometry-aware representations through reconstruction guidance, leading to more accurate denoising in DDPMs. Meanwhile, through analysis and design, SCRG exhibits more effective and lightweight, while decoupled in inference, avoiding computational overhead. Furthermore, we construct two high-quality Text-LiDAR benchmarks ($>$100K samples) using a generalized strategy of geometric annotations, along with a controllability metric. Moreover, a directional position prior is designed to mitigate street distortion, further improving scene fidelity. Additionally, T2LDM++ supports multiple conditions, including (Semantic, Box, BEV, Camera)-to-LiDAR, Sparse-to-Dense, and Dense-to-Sparse generation, by learning a control encoder via frozen DN. With effective prior modeling and high-quality Text-LiDAR benchmarks, T2LDM++ can generate realistic LiDAR scenes with rich geometric details in unconditional and conditional settings.
NUS