Search papers, labs, and topics across Lattice.
This paper introduces LLaVAFlow, an information-theoretic distillation framework designed to mitigate catastrophic forgetting in Multimodal Large Language Models (MLLMs) during visual instruction tuning. By focusing on preserving the cross-modal alignment flow embedded in the information-compression trajectory, LLaVAFlow enhances the transfer of compact alignment information between pretrained and fine-tuned models. Experimental results demonstrate that LLaVAFlow significantly improves downstream task performance and generalization compared to existing methods that overlook this alignment aspect.
Preserving cross-modal alignment flow can dramatically enhance the performance of multimodal models, countering the effects of catastrophic forgetting during fine-tuning.
While Multimodal Large Language Models (MLLMs) exhibit strong generalization, visual instruction tuning for downstream tasks inevitably causes catastrophic forgetting, impairing overall generalization. While existing methods regulate weight updates to reduce forgetting, they overlook the fundamental cross-modal alignment in MLLMs. Based on prior work and our observations, we argue that cross-modal alignment is implicitly captured in the information-compression trajectory. To preserve the alignment flow embedded in the trajectory, we propose LLaVAFlow, an information-theoretic distillation framework. First, we compress the mutual information between the extracted relations and MLLM embeddings, encouraging a learnable module to produce a refined alignment flow that benefits downstream tasks. Second, we maximize the mutual information between the extracted alignment flows of the pretrained and fine-tuned MLLMs, enabling the transfer of compact alignment information. Extensive experiments show that LLaVAFlow is an effective plug-and-play framework that preserves alignment flow and enhances both downstream performance and generalization.
