Search papers, labs, and topics across Lattice.
Cubic Discrete Diffusion (CubiD) is introduced as the first discrete generation model capable of operating on high-dimensional (768-1024 dims) visual representation tokens, overcoming limitations of prior methods restricted to low-dimensional latent spaces. CubiD achieves this by employing fine-grained masking across dimensions and spatial positions, enabling the model to learn intricate correlations within the high-dimensional discrete representation. Experiments on ImageNet-256 demonstrate state-of-the-art discrete generation performance and validate that the generated discrete tokens retain the original representation's capabilities for downstream understanding tasks.
High-dimensional discrete tokens, previously out of reach for generative models, can now be directly generated, unlocking a unified token prediction paradigm for multimodal architectures.
Visual generation with discrete tokens has gained significant attention as it enables a unified token prediction paradigm shared with language models, promising seamless multimodal architectures. However, current discrete generation methods remain limited to low-dimensional latent tokens (typically 8-32 dims), sacrificing the semantic richness essential for understanding. While high-dimensional pretrained representations (768-1024 dims) could bridge this gap, their discrete generation poses fundamental challenges. In this paper, we present Cubic Discrete Diffusion (CubiD), the first discrete generation model for high-dimensional representations. CubiD performs fine-grained masking throughout the high-dimensional discrete representation -- any dimension at any position can be masked and predicted from partial observations. This enables the model to learn rich correlations both within and across spatial positions, with the number of generation steps fixed at $T$ regardless of feature dimensionality, where $T \ll hwd$. On ImageNet-256, CubiD achieves state-of-the-art discrete generation with strong scaling behavior from 900M to 3.7B parameters. Crucially, we validate that these discretized tokens preserve original representation capabilities, demonstrating that the same discrete tokens can effectively serve both understanding and generation tasks. We hope this work will inspire future research toward unified multimodal architectures. Code is available at: https://github.com/YuqingWang1029/CubiD.
