Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces TailLoR, a novel approach for parameter-efficient continual learning that leverages the singular bases of pre-trained weights to establish a fixed reference frame for low-rank updates. By applying a soft spectral penalty, TailLoR minimizes interference from dominant singular directions, allowing for more effective adaptation in long-tail spectral coordinates. The key result demonstrates that this method significantly enhances performance in continual learning scenarios by preserving the integrity of principal components while enabling fine-grained updates.
TailLoR achieves superior continual learning performance by protecting principal components, allowing models to adapt without catastrophic forgetting.
Parameter-efficient finetuning methods based on spectral decomposition have enabled progress in Continual Learning. In this paper we introduce TailLoR, which utilizes the singular bases U and V of the pre-trained weights as a fixed reference frame to learn a low-rank update applied to the singular value matrix. A soft spectral penalty discourages updates aligned with dominant singular directions, reducing interference while routing fine-grained adaptation into the highly flexible, long-tail spectral coordinates.