Search papers, labs, and topics across Lattice.
This paper introduces Distribution-Aware Expert Routing (DAER), a novel framework designed for the continual adaptation of video multimodal large language models (Video-MLLMs) to evolving domains without catastrophic forgetting. By maintaining domain-isolated lightweight experts and employing a distribution-aware routing mechanism, DAER effectively decouples domain-specific knowledge from the pretrained model's general capabilities. Experimental results demonstrate that DAER significantly outperforms existing continual learning methods on a benchmark derived from ten diverse VidQA datasets, highlighting its robustness in handling real-world video data streams.
DAER enables Video-MLLMs to adapt to new domains without sacrificing previously learned knowledge, outperforming traditional methods that struggle with cross-domain interference.
Video multimodal large language models have shown strong capability in video understanding, yet their adaptation to sequentially evolving domains remains underexplored. In real-world deployments, video data often arrives continuously from heterogeneous domains, requiring the model to acquire new domain-specific knowledge without overwriting previously learned capabilities. Existing continual learning methods typically rely on shared adaptation spaces, which can induce severe cross-domain interference and catastrophic forgetting. We propose Distribution-Aware Expert Routing, a parameter-efficient framework for continual Video-MLLM adaptation over evolving domains. DAER maintains domain-isolated lightweight experts while keeping the pretrained Video-MLLM backbone frozen, thereby decoupling domain-specific adaptation from the general multimodal knowledge of the pretrained model. To enable fine-grained specialization, we introduce an intra-domain distribution-aware routing mechanism that matches each input to expert-level prototype reservoirs using MMD. To address the absence of task identities at inference time, we further propose an inter-domain routing mechanism that performs prototype matching in a discriminative subspace for robust domain identification. In addition, we introduce adaptive domain merging to improve parameter scalability and adopt a two-stage optimization strategy to stabilize expert specialization during continual learning. We evaluate DAER by curating a domain-incremental benchmark built from ten VidQA datasets covering diverse visual environments and reasoning demands. Experiments on two strong Video-MLLM backbones show that DAER consistently outperforms prior methods.
