Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper addresses the challenge of long-horizon reinforcement learning by proposing a single-rollout asynchronous training recipe that integrates Muon, token-averaged loss, truncated importance sampling, and auxiliary penalties for length and tool errors. This approach enables efficient policy updates from sparse rewards by allowing learning from ongoing rollouts without waiting for full trajectory completion. Experiments across two model families demonstrate that the full recipe significantly improves training success in repository-level issue resolution compared to baseline methods.
Asynchronous RL training on long-horizon tasks achieves superior performance by leveraging Muon and a novel single-rollout recipe that avoids waiting for complete trajectories.
Long-horizon agents must coordinate reasoning, tool use, and environment feedback over extended interactions. Learning from sparse outcome rewards requires weighting trajectories of different lengths and updating the policy while rollouts are still being collected. We examine a single-rollout asynchronous recipe combining Muon, token-averaged loss, truncated importance sampling, and auxiliary length and tool-error penalties. Single-rollout sampling allows asynchronous learning without waiting for multiple trajectories for the same issue. We study optimizer choices and recipe components for repository-level issue resolution across two model families. The evaluated training runs show higher training success with the full recipe than with the baselines.