Search papers, labs, and topics across Lattice.
This paper investigates catastrophic forgetting in multimodal large language models (MLLMs) during fine-tuning, focusing on visual question answering. They identify two distinct types of forgetting: one caused by out-of-distribution images, mitigated by regularization, and another caused by out-of-distribution text, addressed through a data-hybrid training strategy. The results demonstrate that simple adjustments to fine-tuning recipes can effectively prevent forgetting and even outperform complex continual learning methods, suggesting MLLMs are more robust than previously thought.
MLLMs are surprisingly robust to catastrophic forgetting during fine-tuning, needing only simple regularization or data-hybrid training to maintain performance.
The paper demonstrate that simple adjustments of the fine-tuning recipes of multimodal large language models (MLLM) are sufficient to mitigate catastrophic forgetting. On visual question answering, we design a 2x2 experimental framework to assess model performance across in-distribution and out-of-distribution image and text inputs. Our results show that appropriate regularization, such as constraining the number of trainable parameters or adopting a low learning rate, effectively prevents forgetting when dealing with out-of-distribution images. However, we uncover a distinct form of forgetting in settings with in-distribution images and out-of-distribution text. We attribute this forgetting as task-specific overfitting and address this issue by introducing a data-hybrid training strategy that combines datasets and tasks. Finally, we demonstrate that this approach naturally extends to continual learning, outperforming existing methods with complex auxiliary mechanisms. In general, our findings challenge the prevailing assumptions by highlighting the inherent robustness of MLLMs and providing practical guidelines for adapting them while preserving their general capabilities.
