Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces Replay on Demand (RoD), a method for continued pretraining that dynamically allocates a shared training budget between adaptation and replay samples based on the model's learning dynamics. RoD prioritizes adaptation samples by their remaining learning potential and replay samples by their observed forgetting, creating an emergent curriculum that balances acquiring new knowledge with retaining old. Experiments across various models, scales, and domains demonstrate that RoD achieves or surpasses the adaptation-forgetting trade-off of fixed-replay baselines and model merging without pre-specifying replay mixtures.
Language models can learn new knowledge without forgetting old by dynamically prioritizing replay based on observed forgetting.
Continued pretraining enables language models to adapt to new domains and knowledge, but often at the cost of forgetting previously acquired capabilities. Replay can mitigate this trade-off, but fixed replay mixtures allocate training independently of the model's actual retention needs. We introduce Replay on Demand (RoD), which instead derives the replay allocation from the model's learning dynamics. RoD jointly prioritizes adaptation samples by their remaining learning potential and replay samples by their observed forgetting. Their competition for a shared training budget yields an online curriculum that determines what to train on at each step. Across models, scales, and adaptation domains, RoD reaches or improves upon the adaptation-forgetting frontier of tuned fixed-replay baselines and model merging without prescribing a replay allocation in advance. Replay concentrates on sources that are more vulnerable to forgetting and dynamically increases and redistributes as forgetting emerges during training. Together, our results show that replay can be allocated online from the model's evolving state, targeting what is needed, when it is needed.