Search papers, labs, and topics across Lattice.
This study systematically explores the dynamics of multimodal pretraining, revealing how knowledge flows between language and vision modalities and identifying architectural choices that enhance synergy. Key findings include the importance of early unification in training, which prevents models from defaulting to language priors, and the development of efficient pretraining recipes that significantly reduce compute costs while maintaining performance. The research validates these insights through large-scale experiments with 13.5B MoE models trained on extensive datasets, providing a foundational understanding of multimodal interactions.
Early unification in multimodal training can prevent models from becoming overly reliant on language, unlocking new efficiencies in generative performance.
Vision offers a critical axis for advancing foundation models, driving a shift towards natively unified multimodal pretraining. Despite this momentum, the design space and the fundamental mechanisms of how modalities interact during unified training remain underexplored. We provide empirical clarity through a systematic exploration of multimodal pretraining. Our controlled experiments on both synthetic and large-scale real-world datasets yield four key insights into the physics of multimodal pretraining: (i) Knowledge Flow: We disentangle how language, visual understanding, and visual generation transfer knowledge across modalities, revealing distinct patterns of influence and asymmetry; (ii) Synergy vs. Competition: We show that data"complexity"largely determines whether modalities are synergistic, identify architectural choices that promote synergy: such as shared attention and normalization with modality-specific feed-forward layers, and find that these behaviors generalize across different visual tokenizer designs; (iii) Early Unification: Unifying modalities from the very early stages and training them jointly is shown to be more effective than late alignment or sequential training. This process uncovers a vision laziness phenomenon, where delayed integration leads models to rely on language priors; (iv) Recipes: We derive efficient pretraining recipes that achieve strong generative performance using only 5% of the compute budget. These core findings are subsequently validated at scale by training multiple 13.5B MoE models on 2T tokens. We hope this study provides a principled foundation for understanding and scaling multimodal pretraining.
Meta AI