Search papers, labs, and topics across Lattice.
This paper introduces UniGen-AR, a unified framework that integrates a multi-modal language model with a visual auto-regressive decoder to streamline diverse image generation tasks. By addressing the high inference latency associated with diffusion-based systems, UniGen-AR achieves up to 19 times lower latency while maintaining or enhancing output quality across more than 15 tasks. Key findings highlight the importance of VQ-VAE tokenizer design in optimizing the scalability of visual auto-regressive models for unified visual generation.
Achieving 19 times lower inference latency than diffusion models, UniGen-AR redefines the efficiency of unified visual generation.
Modern computer vision pipelines remain fragmented, with tasks such as text-to-image generation, editing, restoration, and classical perception handled by separate models. We study Unified Visual Generation (UVG), where a single model produces diverse image-valued outputs through a unified multimodal interface. While diffusion-based systems dominate UVG due to strong quality and controllability, their iterative sampling incurs substantial inference latency, limiting practical deployment. To address these limitations, we propose UniGen-AR, a framework that pairs a general-purpose multi-modal language model (MLLM) with an efficient next-scale visual auto-regressive (VAR) decoder. This design retains the flexibility of MLLM-based conditioning while leveraging the sampling efficiency and latent unification properties of VAR models. In our framework, the MLLM encodes free-form instructions and control signals into a unified sequence, which guides the VAR decoder to generate image-valued outputs for over 15 tasks spanning four families. Empirically, UniGen-AR achieves up to $19 \times$ lower inference latency than diffusion-based baselines while maintaining or improving output quality. Our ablations further reveal that VQ-VAE tokenizer design, particularly codebook size and hierarchy, is a critical factor for VAR scalability in UVG. These results establish visual auto-regressive modeling as a compelling and efficient backbone for unified visual generation. Our project page is at https://zpbao.github.io/projects/unigenar.
CMU ML