Search papers, labs, and topics across Lattice.
This paper introduces Observation-Calibrated Self-Distillation (OCSD), a method that enhances token-level supervision in reinforcement learning by contrasting two replay views to isolate the impact of future observations. The authors address the challenge of confounding influences in token scoring, which can obscure the effectiveness of updates derived from privileged information. Experimental results across multiple environments demonstrate that OCSD significantly outperforms existing baselines, providing more accurate alignment with local feedback.
OCSD reveals how isolating observation effects can lead to more effective token-level updates in reinforcement learning, outperforming traditional methods.
Large language model agents are commonly trained through reinforcement learning with sparse trajectory-level rewards, which offer limited guidance on how strongly individual tokens should be updated. On-Policy Self-Distillation (OPSD) addresses this by re-scoring generated tokens under a privileged replay view to obtain dense, token-level supervision. However, we identify a confounding issue: the resulting support may reflect both the privileged information contained in the replay view and score shifts induced by the replay scaffold, making it difficult to attribute the support specifically to that information. This issue is especially pronounced when future environment observations serve as privileged information, since replaying them requires reconstructing an extended scaffold that itself perturbs token scores. To resolve this confounding, we propose Observation-Calibrated Self-Distillation (OCSD), which contrasts two structurally matched replay views, Full and Observation-Ablated, differing only in whether the actual future observation is present, to derive an observation residual that discounts score changes shared by the replay scaffold. OCSD then applies this residual to modulate token-level GRPO updates at high-uncertainty steps, while preserving the trajectory-level update direction. Experiments on ALFWorld, WebShop, and Search-QA across three Qwen3 model scales show that OCSD consistently outperforms strong baselines. Diagnostic analyses further confirm that the calibrated residual aligns better with local environment feedback. Our code is publicly available at https://github.com/yiy1x/OCSD.
