Search papers, labs, and topics across Lattice.
This paper introduces Neural Subspace Reallocation (NSR), which innovatively treats continual learning as a memory management problem over parameter subspaces, utilizing Low-Rank Adaptation (LoRA) modules as retrievable memory units. By employing a cyclic process of compressing learned LoRAs, storing them in a TaskKnowledgeBank, and retrieving them based on similarity for task warm-starting, NSR significantly enhances task recovery efficiency and accuracy. Empirical results demonstrate that NSR reduces cyclic recovery time by 10x and achieves superior performance on benchmark datasets, revealing that the memory mechanism is more critical than learned allocation policies for effective continual learning.
Memory management, not learned policies, is the key to achieving near-zero forgetting in continual learning.
We introduce Neural Subspace Reallocation (NSR), which reframes continual learning as memory management over parameter subspaces. Instead of treating Low-Rank Adaptation (LoRA) modules as disposable per-task adapters, NSR manages them as compressible, retrievable memory units on a frozen backbone through a recurring cycle: (1) compress learned LoRAs via SVD, (2) reserve them in a TaskKnowledgeBank, (3) recall related past LoRAs by embedding similarity to warm-start new or returning tasks, and (4) reallocate the active subspace accordingly, with distillation protecting prior tasks. We prove that in cyclic environments any memoryless allocation policy incurs cumulative regret Omega(T(M-1)Delta_switch) relative to a history-aware policy backed by the Bank (Theorem 1). Empirically, on Split-CIFAR-100 the Bank reduces cyclic recovery time by 10x, exactly as predicted, and on the heterogeneous 5-Datasets benchmark NSR achieves the highest accuracy and the least forgetting, about 9x closer to zero backward transfer than the memoryless heuristics. Crucially, we run a controlled study that isolates which component matters: holding the Bank fixed and varying only the allocation rule, we find that a simple similarity-based retrieval rule matches or beats a learned reinforcement-learning controller (recovering recurring tasks in 0 vs 1.8 steps and reaching equal accuracy). Our central, honest finding is therefore that the memory mechanism -- compression and similarity retrieval -- rather than a learned allocation policy, drives continual-learning performance under fixed capacity. A memory-budget analysis confirms the compressed Bank stays small -- 0.29 MB of parameter memory per task -- so a top-K retention cap bounds the total footprint while preserving fast recovery for retained tasks.
