Search papers, labs, and topics across Lattice.
This paper introduces Geo-LoRA, a geometry-aware framework designed to enhance low-rank adaptation in continual learning by regulating the evolution of shared and task-specific subspaces. By employing techniques such as Subspace Projection Preservation and Adaptive Core-Slack Alignment, Geo-LoRA ensures smooth transitions on the Grassmann manifold, while Median-Calibrated Block Overlap mitigates subspace crowding. The approach consistently outperforms existing methods across various benchmark datasets, demonstrating its effectiveness in maintaining stable representations during class-incremental learning.
Geo-LoRA achieves state-of-the-art performance in continual learning by enforcing geometric constraints that stabilize low-rank adaptations across tasks.
Rehearsal-free class-incremental learning (CIL) with LoRA adapters remains challenging because the low-rank subspaces updated across tasks evolve without geometric control, causing unstable shared representations and repetitive collapse of task-specific updates into previously occupied directions. We introduce Geo-LoRA, a geometry-aware framework that explicitly regulates how low-rank subspaces, both shared and task-specific, evolve during continual learning. For the shared branch, Subspace Projection Preservation (SPP) constrains consecutive updates to follow smooth trajectories on the Grassmann manifold, and Adaptive Core-Slack Alignment (ACSA) decomposes transitions into principal and residual components, aligning the former while modulating the latter to balance stability and plasticity. For the task-specific branch, Median-Calibrated Block Overlap (MCBO) imposes a statistical constraint via normalized projection overlap, penalizing excessive reuse to mitigate subspace crowding. These constraints jointly regulate the evolution of all LoRA subspaces across layers and tasks without introducing additional adapter types beyond standard LoRA. Geo-LoRA provides a principled geometric formulation for continual low-rank adaptation and consistently achieves state-of-the-art performance across multiple benchmark datasets and different task lengths.
