Search papers, labs, and topics across Lattice.
This paper introduces RepFusion, a novel approach that utilizes multimodal large language models (MLLMs) as noisy representation encoders to enhance denoising in text-to-image systems. By leveraging the structured latent space of representation autoencoders (RAEs) and conditioning a diffusion transformer on MLLM outputs, RepFusion significantly improves denoising performance compared to traditional methods using newly initialized denoisers. The findings indicate that MLLMs can effectively serve as strong priors for denoising visual representations, optimizing computational resources during inference.
RepFusion reveals that multimodal large language models can dramatically enhance denoising in text-to-image systems, outperforming traditional denoising methods.
Large language models (LLMs) are widely used in text-to-image (T2I) systems, but they are typically limited to text encoding, while denoising is handled by newly trained generative backbones. The emergence of representation autoencoders (RAEs) shifts the generation target toward semantically structured visual representations, creating a latent space that is more compatible with pretrained LLM priors. Inspired by multimodal LLMs (MLLMs), where an MLP projector is sufficient to align clean visual representations with a pretrained LLM, we repurpose the MLLM itself as a noisy representation encoder, extending this mechanism from clean to noisy inputs. We present RepFusion, which uses the resulting MLLM outputs as the conditioning signal for a diffusion transformer. In controlled comparisons at similar inference budgets, RepFusion outperforms baselines that devote comparable capacity to newly initialized denoisers. These results demonstrate that MLLMs provide strong priors for denoising visual representations and that, by conditioning on evolving noisy representations, test-time compute can be productively spent on repeated MLLM conditioning in modern T2I systems.
Meta AI