Search papers, labs, and topics across Lattice.
This paper introduces Spaced Repetition Training (SRT), a continual learning framework that optimizes the review scheduling of examples during the continual pre-training of language models, inspired by cognitive science principles. By employing the SuperMemo-2 algorithm, SRT adapts the training loop to selectively replay historical examples based on their retention needs, significantly enhancing the stability-plasticity trade-off. The results demonstrate that SRT recovers 5 to 37 percentage points of old-knowledge accuracy lost in naive continual pre-training while maintaining or improving new-knowledge acquisition across various model scales and datasets.
SRT recovers up to 37% of lost knowledge in language models while enhancing new information retention, challenging the limitations of traditional replay methods.
Continual pre-training of large language models must acquire new information without erasing old knowledge. Existing replay methods often choose a global old/new mixture and sample uniformly, ignoring that examples differ in how quickly they are forgotten. We formulate continual pre-training as adaptive review scheduling: the training loop should decide not only how much history to replay, but which examples should return at each step. We introduce Spaced Repetition Training (SRT), a continual learning framework inspired by cognitive science, which schedules sample-rehearsal using the SuperMemo-2 (SM-2) algorithm. SRT maintains per-example review state, maps per-example perplexity to a recall-quality signal, and schedules historical examples for retention and new examples for consolidation while leaving the model, objective, and optimizer unchanged. On temporally separated Wikipedia and code corpora, SRT improves the stability-plasticity trade-off, recovering 5 to 37 percentage points of old-knowledge accuracy lost by naive continual pre-training across model scales while preserving or improving new-knowledge acquisition. At larger scale, SRT preserves broad benchmark performance that naive continual pre-training and uniform replay substantially degrade. Experiments with vision and tabular data further suggest that the scheduling principle extends beyond language when paired with an appropriate recall signal.
