Search papers, labs, and topics across Lattice.
This study evaluates the performance of Muon compared to AdamW in sparse-reward agentic reinforcement learning, specifically within the ALFWorld environment using the Qwen2.5-0.5B-Instruct model. The findings reveal that applying Muon to hidden weight matrices significantly enhances validation success rates, achieving an increase from 0.290 to 0.546 under Group-in-Group Policy Optimization (GiGPO). Additionally, the results indicate that the effectiveness of Muon is sensitive to the choice of advantage estimator and learning rate, with notable improvements observed at lower learning rates.
Muon can boost agentic RL performance by up to 88% compared to AdamW, challenging assumptions about optimizer efficacy in sparse-reward settings.
Muon is competitive with AdamW in large-scale pre-training, but its value for reinforcement-learning (RL) post-training remains unclear. We study vanilla Muon in sparse-reward agentic RL through matched single-seed comparisons with AdamW on ALFWorld using Qwen2.5-0.5B-Instruct. Under Group-in-Group Policy Optimization (GiGPO), applying Muon only to hidden weight matrices raises final-window validation success from 0.290 to 0.546 (+88%); high-rate AdamW controls retain no post-update success. The effect depends on the advantage estimator and learning rate. At 3e-5, Muon improves GRPO from 0.161 to 0.268, whereas GraphGPO's late-window gap narrows near saturation. At 1e-5, GraphGPO Muon reaches 0.901, raises normalized validation AUC from 0.399 to 0.556, and reaches 0.5 and 0.75 success 30 and 60 updates earlier, respectively. These exploratory results show that Muon can benefit agentic RL and motivate studying the policy optimizer, advantage estimator, and learning rate jointly. Multi-seed and cross-task validation remain open.
