Search papers, labs, and topics across Lattice.
This work bridges generative diffusion and self-supervised spectral representation learning through a unified perturbation-kernel framework, resolving why hidden-state alignment accelerates diffusion training. The authors prove that optimizing for spectral representation alignment is mathematically equivalent to performing diffusion score distillation within the network's latent representation space. Applying this insight via a plug-and-play spectral regularizer yields consistent empirical generation gains across both 2D image and 3D point cloud synthesis tasks.
Hidden-state representation alignment in diffusion models isn't just an empirical heuristic鈥攊t is mathematically equivalent to score distillation operating directly within the latent representation space.
Diffusion models have shown remarkable performance on diverse generation tasks. Recent work finds that imposing representation alignment on the hidden states of diffusion networks can both facilitate training convergence and enhance sampling quality, yet the mechanism driving this synergy remains insufficiently understood. In this paper, we investigate the connection between self-supervised spectral representation learning and diffusion generative models through a shared perspective on perturbation kernels. On the diffusion side, samples (e.g., images, videos) are produced by reversing a stochastic noise-injection process specified by Gaussian kernels; on the spectral representation side, spectral embeddings emerge from contrasting positive and negative relations induced by random perturbation kernels. Motivated by this, we propose a self-supervised spectral representation alignment method to facilitate diffusion model training. In addition, we clarify how joint spectral learning can benefit diffusion training from a geometric perspective. Furthermore, we find that the optimization of the spectral alignment objective is in an equivalent form of diffusion score distillation in the representation space. Building on these findings, we integrate a spectral regularizer into diffusion training objectives to improve the performance of diffusion models on multiple datasets. Experiments across images and 3D point clouds show consistent gains in generation quality. Code is released at https://github.com/yuehaowang/spectral-reg-diffusion.
