Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper provides a comprehensive, standardized evaluation of ten self-distillation methods for agentic reinforcement learning against a GRPO baseline across four diverse environments and three model sizes, addressing critical limitations in prior work concerning experimental condition consistency, direct comparisons, training stability, and computational cost. The extensive empirical study, involving 132 training runs and over 12,000 GPU hours, reveals that while self-distillation methods offer some gains, no single method consistently outperforms the baseline, and high final performance can mask significant training instability. Furthermore, increased training cost does not reliably correlate with improved agent performance, highlighting the need for careful consideration of practical trade-offs.
Most self-distillation methods for agentic RL fail to consistently outperform a strong baseline, and impressive final scores often hide unstable training dynamics and substantial computational overhead.
Self-distillation in agentic reinforcement learning has produced a growing number of methods. However, existing evaluations often reuse published baseline scores under differing experimental conditions, provide few direct comparisons among contemporaneous methods, and give limited attention to training stability and computational cost. These limitations obscure the source of reported gains and the practical strengths and weaknesses of competing methods. Costly agent training further motivates a common basis for comparison. To address these gaps, we compare ten representative methods against GRPO on ALFWorld, ScienceWorld, Search-QA, and WebShop across three model sizes. Our study comprises 132 training runs totaling 12,124.4 recorded H800 GPU-hours, with repeated evaluations over eight rollout seeds for the pretrained backbones and all eleven trained methods. Beyond final task performance, we examine training progress, stability, and measured computational cost. Our evaluation yields three key findings. First, gains over GRPO remain limited in scope, with no method consistently outperforming it across settings. Second, strong final performance can mask unstable training. Third, additional training cost does not consistently translate into better performance. We establish this benchmark to support fair, comprehensive comparison and guide future research toward reliable and cost-effective improvements in agent training. Project page: https://aim-uofa.github.io/Self-Distillation-AgentRL/