Search papers, labs, and topics across Lattice.
This paper introduces MMDiff, a framework that enhances diffusion transformers by enabling multi-modal generation of images and dense perceptual modalities through the use of lightweight decoder heads. The authors reveal that perceptual information is distributed across the denoising trajectory, and they implement multi-timestep feature fusion with spatially varying aggregation weights, resulting in a significant 28.7% improvement in semantic segmentation performance compared to single-timestep methods. Additionally, MMDiff demonstrates that frozen diffusion features can effectively compete with advanced encoders like DINOv3, while maintaining strong performance in tasks such as semantic segmentation, salient object detection, and depth estimation.
Multi-modal generation from diffusion transformers can boost semantic segmentation performance by nearly 30% by leveraging temporal perceptual information.
Diffusion transformers have demonstrated remarkable generative capabilities, yet the rich perceptual representations computed across their denoising trajectory are discarded once the content is rendered. We present MMDiff, a framework that transforms a frozen diffusion transformer into a multi-modal generative system that jointly produces images alongside any combination of dense perceptual modalities using lightweight decoder heads. Our central finding is that perceptual information is temporally distributed along the denoising trajectory, and that multi-timestep feature fusion with spatially varying aggregation weights is essential, improving semantic segmentation results by up to 28.7% mIoU over single-timestep extraction. We further adopt concept-driven attention extraction for interpretable spatial guidance, and show that frozen diffusion features are competitive with and complementary to state-of-the-art encoders such as DINOv3. By training only lightweight decoder heads on a frozen backbone, we achieve strong performance in semantic segmentation, salient object detection, and depth estimation, and demonstrate that this framework enables effective synthetic data generation at scale.
