Search papers, labs, and topics across Lattice.
This paper explores Pixel-Space Diffusion Transformers (pDiTs), which directly model raw pixels for high-resolution image synthesis, overcoming limitations of traditional latent diffusion models that rely on fixed visual tokenizers. By removing the VAE bottleneck, pDiTs enable end-to-end optimization and address challenges in high-dimensional modeling, such as noise scheduling and loss weighting. The authors highlight the potential of pDiTs for unified multimodal systems, allowing for the simultaneous processing of raw pixels, text, and task conditions within a single Transformer framework.
Pixel-Space Diffusion Transformers could redefine high-fidelity image generation by integrating visual understanding and generation in a unified model.
Latent diffusion models (LDMs) enable efficient high-resolution image synthesis by denoising in a VAE-compressed latent space. However, fixed visual tokenizers can discard fine textures and structural details, while separate representation and diffusion training creates a mismatch between reconstruction and generation objectives. These limitations have renewed interest in pixel-space diffusion, which models raw pixels directly, removes the VAE bottleneck, and supports end-to-end optimization. This formulation better matches the demands of high-fidelity generation but introduces challenges in high-dimensional modeling, including noise scheduling, loss weighting, token efficiency, and scalable architecture design. Pixel-space modeling also offers a promising basis for unified multimodal systems: raw pixels, text, and task conditions can be represented in a shared token space and jointly processed by a single Transformer, narrowing the gap between visual understanding and generation. This paper reviews Pixel-Space Diffusion Transformers (pDiTs) from the perspectives of model architecture, continuous generative mechanisms, and unified multimodal modeling. We summarize representative methods, identify key technical challenges, and discuss future directions toward high-fidelity, end-to-end vision foundation models that integrate generation and understanding.
