Search papers, labs, and topics across Lattice.
This paper addresses the "Orthogonality Dilemma" in continual learning for LLMs, where strict parameter isolation via orthogonal projections hinders shared representation transfer between related tasks. The authors introduce CoDe-LoRA, a replay-free method that disentangles learning into universal knowledge consolidation and task-specific decoupling using adaptive null space projection and semantic routing. CoDe-LoRA achieves state-of-the-art performance across multiple benchmarks and backbones by effectively balancing knowledge accumulation with task adaptation.
Strict orthogonal parameter isolation in LLM continual learning paradoxically harms knowledge transfer; CoDe-LoRA breaks this dilemma by decoupling universal and task-specific knowledge.
Continual learning (CL) is essential for Large Language Models (LLMs) to sequentially adapt to evolving tasks. To mitigate catastrophic forgetting, recent advances implement low-rank adaptation with orthogonal projections (e.g., O-LoRA) to isolate task parameters. However, we reveal that such strict geometric constraints trigger an"Orthogonality Dilemma": rigid parameter isolation impedes the transfer and accumulation of shared representations across semantically related tasks. In this work, we propose a new replay-free method, called Consolidation and Decoupling LoRA (CoDe-LoRA), for CL of LLMs. CoDe-LoRA disentangles the learning process into Consolidating Universal Knowledge and Decoupling Task-Specific Knowledge. To achieve this, CoDe-LoRA leverages an adaptive null space projection mechanism and semantic routing to balance knowledge accumulation with task-specific adaptation. Experimental results across four backbones and three CL benchmarks show that CoDe-LoRA achieves the best average accuracy. Our code is available at https://github.com/Estrellajer/CoDe-LoRA.
