Search papers, labs, and topics across Lattice.
This paper introduces CRAM, a novel approach for Multimodal Continual Instruction Tuning (MCIT) that addresses the challenges of catastrophic forgetting and parameter efficiency in Multimodal Large Language Models (MLLMs). By isolating task-specific patterns into independent modules and employing adaptive-rank instantiation, CRAM dynamically allocates parameters based on task demands while maintaining stable reuse of existing capabilities. Experimental results across various benchmarks show that CRAM significantly outperforms traditional methods in both retention of learned tasks and efficient parameter utilization.
CRAM achieves a breakthrough in parameter efficiency and task retention by dynamically allocating resources based on task-specific needs, avoiding the pitfalls of both shared updates and isolated expansions.
Multimodal Large Language Models (MLLMs) unify heterogeneous vision-language tasks under a shared generative framework via instruction tuning, yet real-world deployment demands continuous capability expansion, making Multimodal Continual Instruction Tuning (MCIT) essential. Existing methods either update all tasks with a shared parameter set or allocate dedicated modules for each new task. Shared updates force heterogeneous tasks to compete, causing forgetting of learned capabilities. Conversely, isolated expansion prevents interference but severely limits parameter efficiency over long task streams. To address this dilemma, we propose CRAM. Specifically, by isolating task-specific patterns into independent modules, CRAM mitigates catastrophic forgetting across tasks. To further boost parameter efficiency, we utilize adaptive-rank instantiation to identify the capability gap between existing expert capability and new task demands, and dynamically allocate only the necessary parameters. To ensure stable reuse among tasks, centroid-guided routing recognizes and activates existing experts' capabilities, while an orthogonality penalty confines new updates to task-specific directions, preventing re-learning general capability. Extensive experiments across diverse benchmarks consistently demonstrate its superiority over existing methods.
