Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper frames catastrophic forgetting in large pre-trained models as a geometric problem in weight matrix input spaces, revealing gradient-based optimizers' suboptimality for retention. They introduce Local Support Learning (LSL), a framework that augments gradient descent with a gating mechanism to localize updates to specific input distributions without requiring prior data. LSL, implemented with a Gaussian Mixture Model gate, successfully resolves forgetting in LLMs up to 7B parameters, preserving both pre-trained and fine-tuned capabilities efficiently.
A novel gating mechanism allows LLMs to learn new tasks without forgetting old ones, even without access to prior training data.
We explore catastrophic forgetting in the context of large pre-trained models. By considering forgetting as a geometric problem in the input space of each weight matrix, we uncover a natural retention objective under which updates produced by gradient-based optimizers are suboptimal. Following this observation, we propose Local Support Learning (LSL), a general-purpose framework that augments gradient-based training for retention of prior capabilities without access to prior data. During a new learning phase, LSL pairs two components with distinct roles: a standard weight adapter, trained as usual to minimize the loss, and a gating function that enables the adapter only on input activations from its own training distribution, making the update local to that distribution. The key challenge is that this gate must route data from all learning phases while training only on data from the current one. We address this with a gate based on a Gaussian Mixture Model (GMM), whose likelihood decays rapidly away from its training data, giving it a natural tendency to stay closed on data from prior phases. We show that this post-training approach can resolve forgetting in LLMs of up to 7 billion parameters, retaining both pretrained and finetuned capabilities across multiple training phases, while being efficient in memory and compute, robust to hyperparameter choice, and showing scaling potential.