Search papers, labs, and topics across Lattice.
This paper investigates the "modality gap" in MLLMs, where performance degrades when text is presented as images compared to textual tokens, across seven benchmarks and five input modes. Through extensive evaluation and error analysis of 4,000 examples, the authors find that the modality gap is task-dependent and amplified by reading errors (calculation and formatting failures) induced by image rendering, while knowledge and reasoning errors remain relatively stable. To mitigate this, they propose a self-distillation method, training the model on its own text reasoning traces paired with image inputs, achieving a significant accuracy boost on GSM8K (30.71% to 92.72%) and demonstrating transferability.
MLLMs struggle with visually rendered text not because they can't reason, but because they can't *read* it, and a simple self-distillation fix closes the gap.
Multimodal large language models (MLLMs) can process text presented as images, yet they often perform worse than when the same content is provided as textual tokens. We systematically diagnose this"modality gap"by evaluating seven MLLMs across seven benchmarks in five input modes, spanning both synthetically rendered text and realistic document images from arXiv PDFs to Wikipedia pages. We find that the modality gap is task- and data-dependent. For example, math tasks degrade by over 60 points on synthetic renderings, while natural document images often match or exceed text-mode performance. Rendering choices such as font and resolution are strong confounds, with font alone swinging accuracy by up to 47 percentage points. To understand this, we conduct a grounded-theory error analysis of over 4,000 examples, revealing that image mode selectively amplifies reading errors (calculation and formatting failures) while leaving knowledge and reasoning errors largely unchanged, and that some models exhibit a chain-of-thought reasoning collapse under visual input. Motivated by these findings, we propose a self-distillation method that trains the model on its own pure text reasoning traces paired with image inputs, raising image-mode accuracy on GSM8K from 30.71% to 92.72% and transferring to unseen benchmarks without catastrophic forgetting. Overall, our study provides a systematic understanding of the modality gap and suggests a practical path toward improving visual text understanding in multimodal language models.
