Search papers, labs, and topics across Lattice.
This paper introduces Activation-Weighted Adaptive REtention (AWARe), a novel fine-tuning method designed to mitigate catastrophic forgetting in Multimodal Large Language Models (MLLMs). By dynamically controlling parameter updates based on activation patterns, AWARe selectively freezes important parameters that preserve prior knowledge while allowing less critical ones to adapt to new tasks. Experimental results show that AWARe not only maintains upstream capabilities but also enhances downstream performance compared to traditional fine-tuning approaches.
AWARe effectively preserves prior knowledge in MLLMs while boosting performance on new tasks, challenging the conventional trade-off between retention and adaptation.
Multimodal Large Language Models (MLLMs) exhibit strong generalization and reasoning abilities due to large-scale multimodal pre-training. However, fine-tuning these models on downstream tasks often leads to catastrophic forgetting, where newly learned task-specific knowledge degrades previously acquired capabilities. This issue arises because gradient updates for new tasks overwrite parameters critical to prior knowledge, limiting the practical deployment of MLLMs. To address this challenge, we propose Activation-Weighted Adaptive REtention (AWARe), a fine-tuning method that mitigates catastrophic forgetting by dynamically controlling parameter updates based on activation patterns. AWARe assigns activation-based importance scores to parameters, selectively freezing those essential for preserving prior capabilities while allowing less important parameters to adapt to new tasks. Importantly, AWARe operates without modifying model architectures, ensuring compatibility with existing inference engines. Extensive experiments demonstrate that AWARe effectively preserves upstream capabilities while achieving superior downstream performance compared to existing methods. Code is available at https://github.com/kaln27/AWARe.
