Search papers, labs, and topics across Lattice.
Janus-LoRA addresses catastrophic forgetting in LoRA-based continual learning by enforcing orthogonality between LoRA updates and previously learned knowledge subspaces. They achieve this by introducing Gradient Rectification, a closed-form solution that decouples LoRA factor updates, and a Decoupled Margin Loss that promotes feature-level separation. Experiments on continual learning benchmarks demonstrate state-of-the-art performance by harmonizing parameter-level orthogonality with feature-level separation.
LoRA's catastrophic forgetting problem can be solved by explicitly decoupling its factor updates and pushing new feature representations away from old ones, leading to state-of-the-art continual learning performance.
Low-Rank Adaptation (LoRA) has emerged as a promising paradigm for Continual Learning. It independently updates its low-rank factors ($A$ and $B$), creating a composite update to the full weight matrix through their interaction. To prevent catastrophic forgetting, this update should remain orthogonal to the task-specific subspace that contains previously learned knowledge. However, we identify that this composite update systematically violates this orthogonality, reintroducing interference and undermining stability. Furthermore, naively enforcing this orthogonality compromises plasticity, disrupting the delicate stability-plasticity trade-off. To resolve these issues, we propose \textbf{Janus-LoRA}, a framework that restores this balance through two novel components. Specifically, we first introduce Gradient Rectification, a closed-form solution that mathematically decouples LoRA's factor updates, enforcing orthogonality against the historical knowledge subspace identified by an efficient Online Estimation. Next, to enhance plasticity, we introduce a Decoupled Margin Loss that promotes feature-level separation by pushing new feature representations away from old ones, thus creating distinct, low-interference regions for new learning. Comprehensive experiments on challenging benchmarks demonstrate that by harmonizing parameter-level orthogonality with feature-level separation, Janus-LoRA achieves a superior balance and establishes new state-of-the-art performance.
