Search papers, labs, and topics across Lattice.
This paper introduces MileGPO, a novel method for credit assignment in long-horizon reinforcement learning that leverages milestone inference from grouped on-policy rollouts. By employing techniques such as Milestone Discovery, Reliability-Calibrated Shaping, and Progress-Contrastive Calibration, MileGPO effectively identifies and weights significant milestones and traps, enhancing the learning process without requiring additional models or environment interactions. Experimental results demonstrate that MileGPO achieves state-of-the-art performance on ALFWorld and WebShop, with improved robustness to out-of-distribution scenarios.
Milestone inference can significantly enhance credit assignment in long-horizon reinforcement learning, leading to superior agent performance without extra model complexity.
Credit assignment is challenging in long-horizon agentic reinforcement learning, where supervision often comes only from final rewards. Existing methods refine trajectory-level signals into step-level credits through step grouping or graph-based advantage estimation, but can overlook meaningful intermediate milestones. We propose MileGPO (Milestone Inference with Local Evidence for Graph-Based Policy Optimization), which derives process-level credit from grouped on-policy rollouts through three designs. Milestone Discovery identifies candidate milestones on successful rollouts and recurring traps on failed ones. Reliability-Calibrated Shaping (RCS) weights these candidates by outcome-based confidence, strengthening reliable milestones and traps while down-weighting uncertain ones. Progress-Contrastive Calibration (PCC) further tests whether a candidate reflects local progress and whether its incoming ansition outperforms observed alternatives from the same state.MileGPO requires neither auxiliary models nor additional environment interaction. Experiments on ALFWorld and WebShop show state-of-the-art performance and a small in-distribution to out-of-distribution gap on ALFWorld. Ablations and credit diagnostics indicate that reliability weighting, local progress, and same-state branch evidence complement milestone discovery and resolve ambiguous intermediate credit.
