Search papers, labs, and topics across Lattice.
This paper introduces KeepLoRA++, a method for continual learning in pre-trained vision-language models that effectively balances knowledge retention, task adaptation, and plasticity. By analyzing knowledge distribution across Transformer layers, the authors identify that transferable knowledge is primarily in shallow layers, while task-specific adaptations reside in deeper layers. The proposed layer-scaled residual gradient adaptation method enables new tasks to be learned without interfering with previously acquired knowledge, achieving superior performance across multiple tasks compared to existing baselines.
KeepLoRA++ reveals that effective continual learning hinges on strategically managing knowledge across different layers of a model, leading to significant performance gains in complex tasks.
Continual learning for pre-trained vision-language models requires balancing three competing objectives: retaining pre-trained knowledge, preserving knowledge from a sequence of learned tasks, and maintaining the plasticity to acquire new knowledge. This paper presents KeepLoRA++, balancing these objectives through a unified dual-dimensional knowledge retention mechanism. We analyze knowledge distribution of Transformer architecture from both inter-layer and intra-layer perspectives. The inter-layer perspective examines how retention is distributed across layers, while the intra-layer perspective focuses on the parameter space within each layer. Our analysis reveals a structural property: general transferable knowledge is mainly encoded in the shallow layers and the principal subspace of the parameters, while task-specific adaptations are localized in the deep layers and the residual subspace. Motivated by this insight, KeepLoRA++ introduces a layer-scaled residual gradient adaptation method. New tasks are learned by restricting LoRA parameter updates to the residual subspace, combined with a shallow-to-deep layer scaling, to prevent interference with previously acquired capabilities. Specifically, the gradient of a new task is projected onto a subspace orthogonal to both the principal subspace of the pre-trained model and the dominant directions of previous task features, while simultaneously assigning smaller update magnitudes to shallow layers and larger ones to deeper layers. Our theoretical analysis and empirical evaluations confirm that KeepLoRA++ successfully balances these three competing objectives, consistently outperforming representative baselines across image classification, visual question answering, and video understanding tasks.
