Search papers, labs, and topics across Lattice.
This paper introduces ProtoAda, a novel framework for Multimodal Continual Instruction Tuning (MCIT) that utilizes prototype-guided adaptive tuning to enhance task assignment and routing in Multimodal Large Language Models (MLLMs). By incorporating format-aware task prototypes, ProtoAda aligns task semantics with output structures, mitigating the inter-task interference that arises from format-blind task assignments. Experimental results show that ProtoAda significantly outperforms existing methods, particularly in scenarios where answer structures are vulnerable to degradation from sequential tuning.
Format-aware task prototypes in ProtoAda prevent inter-task interference, leading to superior performance in multimodal instruction tuning.
Multimodal Large Language Models (MLLMs) achieve strong performance through instruction tuning, but real-world deployment requires them to continually acquire new vision-language capabilities, making Multimodal Continual Instruction Tuning (MCIT) essential. To reduce inter-task interference and promote collaboration, recent methods often employ sparse architectures like Mixture of LoRA Experts with image-text similarity routing. However, tasks with distinct response structures could share highly similar visual-linguistic semantics and thus be wrongly routed to the same expert; image-text similarity alone is insufficient for reliable task assignment. For example, an expert in a grounding task requiring coordinate prediction may be biased toward producing short textual answers after learning semantically similar VQA tasks. This format-blind task assignment integrates heterogeneous response types into shared parameters, inducing gradient interference and ineffective expert collaboration. To address this problem, we propose ProtoAda, a prototype-guided adaptive tuning framework. ProtoAda introduces format-aware task prototypes to align task assignment and routing with both task semantics and output structure, and further consolidates format-compatible updates in a geometry-aware manner to effectively reuse and progressively refine existing parameters. Extensive experiments on multiple benchmarks demonstrate that ProtoAda achieves superior performance, especially on tasks whose answer structures are easily corrupted by sequential tuning.
