Search papers, labs, and topics across Lattice.
Current text-to-image pipelines bypass explicit semantic reasoning by directly mapping text embeddings to continuous latent diffusion noise, creating an interpretability and alignment bottleneck. VoT resolves this by inserting a discrete visual-thinking layer between Vision-Language Models (VLMs) and Diffusion Transformers (DiTs), turning the VLM into a multimodal planner that autoregressively generates intermediate visual tokens before pixel rendering. By training a dedicated vector-quantized tokenizer with joint alignment and reconstruction objectives, the framework improves text-image semantic consistency while providing an inspectable, controllable planning interface.
Diffusion models no longer need to simultaneously plan and paint; inserting discrete "visual thought" tokens between VLMs and DiTs decouples high-level semantic reasoning from pixel synthesis for tighter alignment and direct intermediate control.
Current text-to-image systems typically employ a "text encoder plus diffusion decoder" paradigm, in which text semantics directly modulate continuous latent noise. Despite their success, these methods lack an explicit, interpretable intermediate representation that effectively bridges high-level linguistic semantics and low-level visual signals. In this paper, we propose Vision-of-Thought (VoT), a framework that introduces a discrete visual-thinking layer between vision-language models (VLMs) and diffusion transformers (DiTs). Instead of treating VLMs merely as text encoders, we use them as multimodal planners that generate discrete VoT tokens representing high-level visual plans, such as objects and layouts, before rendering pixels. We train a specialized VoT tokenizer in the VLM semantic space with a closed-loop objective that combines VLM alignment, feature reconstruction, and vector-quantization losses. These objectives make the tokens semantically readable by the VLM while preserving the visual information needed for generation. Experimental results demonstrate that VoT improves semantic alignment and provides a structured interface for interpretable and controllable generation.
