Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper investigates the necessity of internet-scale language pretraining for text-to-image (T2I) generation by introducing Halcyon, an autoregressive multimodal model pretrained from scratch on 100M image-text pairs. Halcyon-0.52B and Halcyon-1.6B achieve competitive T2I performance against larger, language-pretrained models while requiring significantly less pretraining compute. Controlled experiments confirm that text-only pretraining offers no discernible benefit for T2I generation quality.
State-of-the-art text-to-image generation is achievable with 18x less compute by training from scratch, bypassing expensive language pretraining.
Modern text-to-image (T2I) systems use internet-scale language pretraining for text-conditioned image generation. We ask whether language pretraining is necessary for strong T2I generation. We introduce Halcyon, a family of early-fusion multimodal autoregressive models pretrained entirely from scratch on GPIC, a dataset of 100M image-text pairs. Halcyon-0.52B outperforms LlamaGen-XL/8 (48.0 vs. 63.3 gFD) with 18.2times less pretraining compute. Further, Halcyon-1.6B outperforms JiT-T2I-XXL/8++ (35.3 vs. 41.0 gFD) with 275times less pretraining compute. Controlled experiments show that text-only pretraining does not improve T2I generation. Halcyon also learns image captioning by replacing caption dropout with I2T supervision, maintaining competitive T2I generation quality under the same pretraining budget. We show that strong T2I systems can be trained from scratch without internet-scale language pretraining. Code and models are available at https://cs.stanford.edu/~keshik/halcyon
Stanford HAI