Search papers, labs, and topics across Lattice.
This paper introduces a Visual Dependence-Aware (VDA) framework for Multimodal Unsupervised Continual Post-Training (MU-CPT) that addresses the limitations of existing methods by incorporating visual dependence (VD) into the learning process. By utilizing Visually Constrained Optimal Transport (VC-OT) to manage the structural distortion of VD and Visually Modulated Adaptation (VMA) to enhance new-task learning, the framework effectively mitigates cross-modal catastrophic forgetting while promoting adaptability. Experimental results demonstrate that VDA successfully balances the stability of old tasks with the plasticity required for new tasks, marking a significant advancement in continual learning for multimodal large language models (MLLMs).
Token-level visual dependence is the key to preventing catastrophic forgetting in multimodal continual learning, enabling models to adapt without losing prior knowledge.
In this paper, we explore a novel task of Multimodal Unsupervised Continual Post-Training (MU-CPT), enabling deployed MLLMs to continually evolve from streaming unlabeled data. Existing unsupervised post-training methods for MLLMs typically optimize target tokens uniformly, overlooking their heterogeneous visual dependence (VD). However, we reveal that token-level VD is crucial for MU-CPT. Specifically, its structural distortion serves as an indicator of cross-modal catastrophic forgetting, and its inherent heterogeneity acts as a compass to guide new-task learning. Leveraging this property, we propose a Visual Dependence-Aware (VDA) framework with two main components. First, Visually Constrained Optimal Transport (VC-OT) formulates the VD structural distortion of old-task VD during new-task learning as an optimal transport problem to mitigate cross-modal forgetting. By designing a region-aware ground cost and a dependence-stratified transport penalty, it prevents global shifts in visual focus while strictly prohibiting visual reliance from degenerating into language bias. Second, Visually Modulated Adaptation (VMA) exploits VD heterogeneity to emphasize visually grounded new-task learning, promoting new-task plasticity. Together, our method simultaneously maintains old-task stability and new-task plasticity during challenging MU-CPT. Extensive experiments under our MU-CPT setting validate the effectiveness of VDA.
