Search papers, labs, and topics across Lattice.
This paper introduces ViGOS, a visually grounded on-policy self-distillation framework designed to enhance multimodal large language models (MLLMs) by decoupling perception from reasoning. By employing a two-teacher system鈥攐ne for visual description and another for reasoning鈥擵iGOS mitigates the risk of shortcut learning that can arise when models overly rely on textual references. Experimental results across various benchmarks demonstrate that ViGOS not only retains the advantages of traditional OPSD but also significantly improves image-grounded performance in scenarios prone to shortcuts.
ViGOS reveals that decoupling perception from reasoning can significantly enhance multimodal model robustness against shortcut learning.
On-policy self-distillation (OPSD) trains a model on its own rollouts and uses a frozen copy to provide dense token-level targets conditioned on a reference target. This works well for LLM reasoning, but a direct extension to multimodal large language models (MLLMs) can create a shortcut: the privileged target may guide tokens mainly based on the text reference target rather than the image. We propose ViGOS, a visually grounded OPSD framework for MLLM post-training. The student first writes a visual description and then reasons toward the final answer. For valid rollouts, an image-only perception teacher supervises the description, while a privileged reasoning teacher supervises the reasoning and final answer on the same student prefix. A reference teacher is used only for invalid rollouts to recover the output format. Across general vision-language, expert reasoning, visual math, spatial grounding, and visual-language-prior benchmarks, ViGOS keeps the main benefits of OPSD and improves image-grounded behavior in shortcut-prone settings.
