Search papers, labs, and topics across Lattice.
This paper explores the potential of Visual Pretraining as a method for enhancing language intelligence by directly utilizing visually rich documents instead of converting them to text. The authors demonstrate that this approach consistently outperforms traditional text-only pretraining across various model architectures and evaluation benchmarks. The findings suggest that incorporating visual information can significantly improve the performance of foundation models, challenging the prevailing reliance on text-based training methods.
Visual Pretraining outperforms text-only methods, revealing that rich visual cues can enhance language model performance in ways previously underestimated.
The rapid progress of large foundation models has been driven predominantly by pretraining on large-scale text corpora. However, many forms of knowledge are conveyed through visual representations, where figures, typeset equations, and page layouts carry rich information that cannot be faithfully or completely captured by text alone. Yet current pretraining approaches discard these visual cues by converting visually rich sources, such as documents and web pages, into plain text for learning language intelligence. This paper challenges the default assumption that language models must be trained on text-only representations and shows that Visual Pretraining is a scalable learner for foundation model intelligence. To this end, we conduct a systematic study of unsupervised visual pretraining paradigms that directly leverage visual documents without text extraction. Across multiple backbones and benchmarks, visual pretraining on the same underlying corpora consistently outperforms text-only pretraining, offering an efficient pathway to scalable language intelligence.
