Search papers, labs, and topics across Lattice.
This paper introduces Golden-GRPO Injection (GRIN), a mixed-policy reinforcement learning framework designed to enhance continual knowledge injection in large language models. Unlike traditional supervised fine-tuning, which struggles with generalization across varied contexts, GRIN employs a three-stage self-learning approach that leverages a golden answer to provide effective learning signals even when encountering novel facts. Experimental results demonstrate that GRIN significantly outperforms supervised fine-tuning and other mixed-policy RL baselines, particularly in complex reasoning tasks, while maintaining performance on basic fact recall.
Mixed-policy reinforcement learning can enable language models to absorb knowledge more effectively than traditional supervised fine-tuning, especially in complex reasoning scenarios.
Continual knowledge injection is essential for keeping large language models up-to-date in a fast-evolving world. Existing methods rely on supervised fine-tuning (SFT), which memorizes injected facts in their training format but fails to generalize across paraphrasing, document combinations, and reasoning. To address this, we propose Golden-GRPO Injection (GRIN), a three-stage self-learning framework for continual knowledge injection. Golden-GRPO is a mixed-policy reinforcement learning algorithm designed specifically for knowledge injection, which injects a golden answer to provide learning signal even when on-policy rollouts fail on novel facts. We further introduce Blank and Counter, two document-level benchmarks targeting novel acquisition and counterfactual overwrite respectively, each evaluating single-fact recall, multi-source retrieval, and inferential reasoning. Our experiments establish a clear empirical claim: mixed-policy reinforcement learning enables knowledge absorption beyond what supervised fine-tuning can achieve. GRIN substantially outperforms SFT and mixed-policy RL baselines on the harder question types while matching them on basic fact recall.
