Search papers, labs, and topics across Lattice.
This paper provides a novel function-space perspective on catastrophic forgetting in continual adaptation, revealing that forgetting is concentrated in a limited number of output-space directions. By analyzing the neural tangent kernel (NTK) regime, the authors derive a closed-form predictor for the forgetting vector that accurately captures old-task prediction drift during new-task training. Their findings indicate that the vulnerable rank of forgetting can be characterized by a Kronecker scaling rule, highlighting the limitations of traditional parameter-space approaches and suggesting a targeted spectral regularizer to mitigate forgetting.
Forgetting in continual learning is not random; it clusters in a few critical output-space directions, revealing a surprising structure in how models adapt to new tasks.
Catastrophic forgetting in continual adaptation is usually studied through parameter drift, replay, or distillation, but these views do not identify which output-space directions are vulnerable. We give a function-space account in the NTK regime: new-task training induces old-task prediction drift through the cross-task kernel, yielding a closed-form predictor for the forgetting vector before any new-task gradient step. In frozen-backbone linear-head PEFT-CL, where the model is linear in the trainable parameters, the predictor is exact up to numerical precision; for nonlinear adapters/full fine-tuning, it is a local NTK approximation. The same expression reveals that forgetting concentrates in a small number of old-task NTK eigenmodes and under frozen linear heads gives a Kronecker scaling rule for the vulnerable rank. These results clarify the relation to prior NTK-overlap theory, explain why parameter-space regularizers can miss output-space interference, and motivate a targeted spectral regularizer.
