Search papers, labs, and topics across Lattice.
The authors train a LiDAR-conditioned, multi-task diffusion model on pseudo-labels from 2D foundation models and back-project intermediate UNet activations onto the sparse point cloud, strictly excluding spatial coordinates to isolate learned representations. Linear probes on these coordinate-free point features recover ~23% mIoU on 3D semantic segmentation (versus ~3.5% random control), with feature-similarity analyses revealing that task streams converge into a shared cross-modal bottleneck before re-specializing in the decoder. This validates that conditional diffusion can serve as an effective representational bridge, transferring large-scale 2D visual priors directly into structured 3D domains without native 3D supervision.
2D foundation model priors can be transferred directly into coordinate-free 3D point representations via conditional diffusion, forming an emergent multimodal bottleneck that achieves non-trivial 3D semantic understanding without native 3D labels.
Transferring the rich priors of large 2D foundation models to sparse 3D LiDAR remains challenging, as training native 3D foundation models at comparable scale is limited by data and annotation scarcity. We introduce a LiDAR-conditioned diffusion model trained on pseudo-labels from off-the-shelf 2D foundation models. The model supports multiple output modalities, including depth, semantic segmentation and instance prediction, selectable via a textual task prompt. Because the model is conditioned on LiDAR, both its outputs and its intermediate UNet features can be projected back onto the input point cloud, enabling analysis of a 3D representation learned entirely under 2D supervision. We study this representation directly in point-cloud space, explicitly excluding raw spatial coordinates to isolate feature content from projection geometry. Linear probes recover up to ~23% Mean Intersection over Union (MIoU) on 3D semantic classes, compared to ~3.5% for a matched Gaussian-noise control, indicating substantial non-trivial structure. Pairwise cosine similarity across modality-specific feature streams reveals a layered organization. Early encoder layers remain weakly aligned across modalities while individually decodable, intermediate layers converge toward a shared representation, and decoder layers re-specialize toward task-specific outputs. These findings indicate that LiDAR-conditioned diffusion models can induce structured 3D representations from 2D supervision alone, with a modality-dependent manifold that locally unifies near a shared bottleneck. This positions diffusion as a viable mechanism for transferring large-scale 2D priors into sparse 3D domains.
