Search papers, labs, and topics across Lattice.
The paper introduces Distillation via Split Contexts (DiSC), a context-distillation method for continually adapting LLMs to new knowledge while mitigating forgetting of previously learned capabilities. DiSC derives student and teacher distributions by conditioning on distinct segments of the training example and minimizes the KL divergence between the shared tokens, avoiding explicit generation steps. Experiments on four post-trained models and two adaptation domains demonstrate that DiSC achieves a better trade-off between learning new knowledge and retaining skills like instruction-following, reasoning, and factual knowledge compared to existing fine-tuning and distillation methods.
LLMs can learn new information without forgetting old tricks using a surprisingly simple context distillation method.
Post-training endows pretrained LLMs with a variety of desirable skills, including instruction-following, reasoning, and others. However, these post-trained LLMs only encode knowledge up to a cut-off date, necessitating continual adaptation. Unfortunately, existing solutions cannot simultaneously learn new knowledge from an adaptation document corpora and mitigate the forgetting of earlier learned capabilities. To address this, we introduce Distillation via Split Contexts (DiSC), a simple context-distillation based approach for continual knowledge adaptation. \methodname~derives student and teacher distributions by conditioning on distinct segments of the training example and minimizes the KL divergence between the shared tokens. This allows us to efficiently apply context-distillation without requiring explicit generation steps during training. We run experiments on four post-trained models and two adaptation domains. Compared to prior finetuning and distillation methods for continual adaptation, DiSC consistently reports the best trade-off between learning new knowledge and mitigating forgetting of previously learned skills like instruction-following, reasoning, and factual knowledge.
