Search papers, labs, and topics across Lattice.
This paper introduces InduceKV, a retrieval-based method for fixed-footprint continual adaptation of multimodal large language models that maintains a consistent memory budget while allowing for task-specific updates. By employing a bilevel selection process to create a compact inducing set of attention-ready memory entries, InduceKV effectively balances current-task likelihood and retention while ensuring coverage in the retrieval space. Experimental results demonstrate that InduceKV outperforms existing methods such as PEFT, MoE, and replay under equivalent memory constraints across various tasks, indicating its robustness and efficiency in continual learning scenarios.
InduceKV achieves superior performance in continual adaptation of multimodal LLMs while strictly adhering to a fixed memory budget, challenging the notion that larger memory footprints are necessary for effective learning.
Multimodal large language models must adapt to evolving tasks and domains, yet continual improvement under bounded deployment footprint remains difficult because repeated parameter updates or growing replay stores can accumulate adaptation state over time. We study fixed-footprint continual adaptation: the deployed adaptation state is kept under a fixed memory budget, while the backbone model is left unchanged and task-specific updates are externalized. We propose InduceKV, a retrieval-based method that stores each selected training prefix as an attention-ready memory entry, consisting of a frozen retrieval key and compact layerwise key--value (KV) payloads that can be appended to the model's self-attention cache. Under a strict memory budget, InduceKV constructs a compact inducing set through bilevel selection: a lightweight calibration is fit for retrieval, while the selected memory balances current-task likelihood, anchor-based retention, and coverage in the frozen retrieval space. Across task-incremental instruction tuning, continual VQA, domain-incremental adaptation, and lifelong multimodal instruction tuning, InduceKV consistently improves over PEFT, MoE, replay, and prompt-retrieval baselines under matched memory budgets. We further report backbone-matched, stage-1 CoIN, compute-matched, and scalability diagnostics, showing that the gains are not due to a stronger backbone, replay alone, or an unbounded candidate pool.
