Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
To mitigate catastrophic forgetting in multimodal continual instruction tuning without relying strictly on rigid parameter isolation, the authors introduce ReCAP, a retrieval-guided framework that orchestrates modular capability reuse via an external knowledge base. The system dynamically assembles instance-specific execution paths of capability modules guided by retrieved reasoning patterns, while introducing adaptive subspace recycling to protect historically critical parameter directions and repurpose residual capacity across sequential tasks. Across multiple continual instruction tuning benchmarks, this knowledge-guided routing and subspace recycling achieves state-of-the-art retention and forward transfer without parameter explosion.
Multimodal continual learning does not require rigid parameter isolation or massive replay buffers—external reasoning retrieval can dynamically compose modular capability paths while subspace recycling safely prevents cross-task interference.
Multimodal continual instruction tuning (MCIT) aims to enable multimodal large language models to acquire new capabilities from sequential tasks while preserving previously learned knowledge. Existing methods primarily mitigate catastrophic forgetting by constraining parameter updates or separating task-specific adaptations. However, continual adaptation can also benefit from external knowledge that provides domain-specific information and reusable reasoning patterns for solving diverse instructions. For example, to answer "How many red cubes are to the left of the sphere?", domain knowledge can provide relevant concepts about objects and spatial relations, while reasoning knowledge can specify ordered operations such as object recognition, spatial filtering, and counting. Despite this potential, how to leverage external knowledge for continual adaptation remains largely unexplored in existing MCIT methods. To this end, we propose ReCAP, a retrieval-guided framework that leverages external knowledge to guide capability reuse during continual adaptation. At each continual stage, ReCAP uses external search and an LLM to incrementally build a knowledge base of domain, reasoning, and format knowledge based on the current-stage training data. For each instruction, retrieved domain knowledge guides generation, while retrieved reasoning knowledge selects and orders capability modules to form an instance-specific capability path. As these capability modules are reused across stages, subsequent adaptation can overwrite previously learned parameters. To enable stable cross-stage reuse, ReCAP introduces adaptive subspace recycling, which parameterizes reusable capability modules with shared bases and stage-specific cores, protects historically important directions while recycling residual capacity. Extensive experiments on MCIT benchmarks show that ReCAP achieves SOTA performance.