Search papers, labs, and topics across Lattice.
This study investigates the scaling properties of native multimodal pre-training for transformer-based vision-language models, revealing that minimal objective loss follows a predictable compute law while optimal model sizes and token counts scale as power laws. The research highlights distinct scaling behaviors for language and multimodal objectives, with the latter being sensitive to data composition, particularly in text-heavy mixtures that require larger models for efficiency. Ultimately, the findings establish a framework for optimizing model configurations, demonstrating that native multimodal pre-training enhances both spatial reasoning and multimodal in-context learning.
Scaling native multimodal pre-training reveals that text-heavy data mixtures require larger models for optimal efficiency, challenging conventional resource allocation strategies.
Although large language models (LLMs) exhibit remarkable reasoning capabilities, their reliance on text-only pre-training restricts the perception of the multimodal physical world. Native multimodal pre-training avoids this limitation by training models from scratch on multimodal inputs, thereby achieving deep cross-modal integration and mitigating optimization asymmetries inherent to traditional late-fusion architectures. Despite these advantages, the scaling properties of this paradigm remain systematically uncharacterized. To address this gap, we investigate the optimal model size and token count for training a transformer-based vision-language model under a fixed computational budget. We demonstrate that minimal objective loss adheres to a predictable compute law, whereas compute-optimal model sizes and token counts scale as power laws. Notably, language and multimodal objectives manifest distinct scaling behaviors. The language allocation law is largely invariant to the composition of the data, indicating stable language learning regardless of the multimodal data ratio. Conversely, the multimodal allocation law is highly sensitive to this composition. Specifically, text-heavy mixtures become compute-efficient only at larger model scales, shifting the optimal resource allocation toward greater model capacity. Additionally, by modeling the influence of data composition on compute laws and allocation exponents, we derive an efficiency frontier specifying precise configurations of model size, token count, and data mixture. Downstream evaluations further reveal that native multimodal pre-training induces positive cross-modal transfer, thereby enhancing pure-text spatial reasoning and enabling robust multimodal in-context learning. In summary, this empirical research establishes the essential groundwork for predictably scaling multimodal foundation models.
