Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper establishes empirical scaling laws comparing encoder-free multimodal large language models—which ingest raw patch tokens directly—against standard encoder-based systems across text and multimodal loss frontiers. Determining if dedicated visual backbones can be eliminated is critical for unifying pretraining pipelines and removing inductive biases inherent to frozen visual priors. The results reveal that while encoder-free architectures underperform at small scales, they achieve parity on multimodal loss around $10^{22}$ FLOPs as the backbone transformer autonomously develops visual specialization in its early layers.
Pretrained visual encoders are a temporary crutch: encoder-free multimodal architectures close the performance gap with encoder-based systems at roughly $10^{22}$ FLOPs, well within current frontier pretraining budgets.
Most modern multimodal large language models (MLLMs) build on a pretrained visual encoder that provides a strong visual prior. Encoder-free MLLMs instead learn visual representations directly from raw pixels, offering a simple and unified architecture, but their scaling behavior has not been systematically characterized. To fill this gap, we compare scaling laws for encoder-free and encoder-based MLLMs and report three main findings: (1) Removing the visual encoder shifts the compute-optimal allocation for the multimodal objective toward larger models, while leaving that for text nearly unchanged. (2) The two architectures exhibit nearly overlapping loss--compute frontiers on the text objective, but diverge on the multimodal objective: encoder-free models underperform at small scales yet are predicted to catch up at around $10^{22}$ FLOPs, well within practical pretraining budgets. (3) Without a visual encoder, the language model learns to take over its role via vision-specific adaptation: bidirectional interactions among visual tokens become increasingly beneficial as training compute grows, visual processing shifts toward earlier layers, and expert routing for visual tokens becomes more concentrated. Overall, our results indicate that the advantage of the visual prior provided by a pretrained encoder diminishes with scale, positioning encoder-free architectures as a promising direction for multimodal pretraining.