Search papers, labs, and topics across Lattice.
This paper addresses the challenge of projector-level forgetting in Multimodal Continual Instruction Tuning (MCIT) by introducing Progressive Multimodal Alignment (PMA), a framework that allows the projector to adapt to shifting visual distributions and evolving instruction semantics while retaining previously learned alignments. PMA employs a lightweight representation descriptor to detect multimodal distribution shifts and utilizes an expandable router to integrate expert outputs, ensuring stability and plasticity with minimal parameter growth. Experimental results across two MCIT benchmarks show that PMA significantly improves performance over existing state-of-the-art methods, highlighting its effectiveness and scalability across various Multimodal Large Language Model (MLLM) backbones.
Projector-level forgetting in MLLMs can be mitigated, leading to significant performance gains in continual instruction tuning without excessive parameter growth.
Multimodal Large Language Models (MLLMs) rely on a projector to align visual representations with the language embedding space, making it central to cross-modal understanding. In Multimodal Continual Instruction Tuning (MCIT), however, shifting visual distributions and evolving instruction semantics cause this shared projector to drift, leading to projector-level forgetting, an issue largely overlooked by methods that focus primarily on the LLM backbone. We introduce Progressive Multimodal Alignment (PMA), a framework that enables the projector to adapt continually while preserving previously learned alignment. PMA detects multimodal distribution shifts via a lightweight representation descriptor and progressively expands projector experts only when needed. An expandable router integrates expert outputs based on multimodal features, while the original pretrained projector is retained as a stable alignment anchor. This progressive mechanism balances stability and plasticity with sub-linear parameter growth and serves as a method-agnostic add-on to existing MCIT approaches. Extensive experiments on two recent MCIT benchmarks demonstrate that mitigating projector-level forgetting yields consistent gains over prior state-of-the-art methods when combined with PMA. Moreover, PMA scales across diverse MLLM backbones, demonstrating robust and broadly applicable MCIT performance.
