Search papers, labs, and topics across Lattice.
This paper introduces Selective Importance Sampling (SIS), a novel approach to improve the alignment of large language models (LLMs) by converting off-policy tokens into on-policy tokens, thus eliminating the need for importance score corrections. By employing a token-level rejection sampling method, SIS effectively reduces variance in gradient estimators and enhances the robustness of LLMs under off-policy training conditions. Experimental results demonstrate that SIS consistently outperforms existing methods across various benchmarks, indicating its potential for broader application in reinforcement learning post-training strategies.
Transforming off-policy tokens into on-policy ones could be the key to unlocking more robust and efficient alignment for large language models.
Reinforcement learning (RL) post-training for large language models (LLMs) follows a efficient paradigm of"rollout then update", which inevitably results in off-policy training data. To resolve this, Importance sampling (IS) is proposed, while the token-level ratios compound over long sequences, causing severe variance exploded. A natural idea is"transferring"these off-policy token into on-policy token, so that the importance scores for correction are unnecessary. Following this idea, we propose Selective Importance Sampling (SIS), which is inspired by rejection sampling. Concretely, SIS implements by viewing off-policy model as proposal distribution, and implement a token-level rejection test: accepted tokens are viewed as on-policy, so that receive unit importance score, while rejected tokens retain the standard IS correction. Our proposed SIS is theoretically proved reducing the gap between token-level and sequence-level off-policy gradient estimators. The SIS acts as a plug-in that only modifies the importance ratio in the policy loss, adding negligible wall-clock overhead, and can be combine with a vast vary of RL post-training algorithms. Experiments on dense and MoE LLMs across math and agent benchmarks show that SIS consistently improves all objectives, while providing substantially stronger robustness under off-policy data.
