Search papers, labs, and topics across Lattice.
The paper introduces Coevolving Representation Diffusion (CoReDi), a method that jointly learns a lightweight linear projection to adapt the semantic representation space used in image-feature diffusion models. By co-optimizing this projection with the diffusion model using stop-gradient targets, normalization, and regularization, CoReDi prevents feature collapse and allows the semantic space to specialize for image synthesis. Experiments demonstrate that CoReDi achieves faster convergence and higher sample quality compared to joint diffusion models with fixed representation spaces in both VAE latent diffusion and pixel-space diffusion settings.
Image diffusion models get a boost by letting their semantic feature space evolve during training, leading to faster convergence and better image quality.
Joint image-feature generative modeling has recently emerged as an effective strategy for improving diffusion training by coupling low-level VAE latents with high-level semantic features extracted from pre-trained visual encoders. However, existing approaches rely on a fixed representation space, constructed independently of the generative objective and kept unchanged during training. We argue that the representation space guiding diffusion should itself adapt to the generative task. To this end, we propose Coevolving Representation Diffusion (CoReDi), a framework in which the semantic representation space evolves during training by learning a lightweight linear projection jointly with the diffusion model. While naively optimizing this projection leads to degenerate solutions, we show that stable coevolution can be achieved through a combination of stop-gradient targets, normalization, and targeted regularization that prevents feature collapse. This formulation enables the semantic space to progressively specialize to the needs of image synthesis, improving its complementarity with image latents. We apply CoReDi to both VAE latent diffusion and pixel-space diffusion, demonstrating that adaptive semantic representations improve generative modeling across both settings. Experiments show that CoReDi achieves faster convergence and higher sample quality compared to joint diffusion models operating in fixed representation spaces.
