Search papers, labs, and topics across Lattice.
This paper introduces Temporal GRPO, a novel approach to reinforcement learning that mitigates trajectory-level credit aliasing by aligning rollouts with specific task stages. By applying stage-specific advantages to corresponding action intervals, the method enhances task success and sample efficiency in vision-language-action policies. Experiments on RoboTwin 2.0 demonstrate consistent performance improvements across varying task horizons, showcasing the effectiveness of controlled updates in preserving prerequisite stages.
Temporal GRPO reveals that aligning reinforcement learning updates with task stages can significantly boost both efficiency and success rates in complex vision-language-action tasks.
Outcome-driven reinforcement learning offers a scalable way to post-train vision-language-action (VLA) policies from sparse task-success feedback. In common GRPO-based VLA post-training, one rollout-level advantage is applied to every action in the trajectory. A rollout that completes several valid stages but fails later can therefore penalize the actions that produced its earlier progress. We call this trajectory-level credit aliasing. Temporal GRPO addresses this problem by constructing detectable task stages, aligning each rollout with stage-specific action intervals, and comparing only rollouts that have entered the same stage. The resulting stage advantages are applied to their corresponding intervals in a single policy update. On RoboTwin 2.0, Temporal GRPO improves task success and sample efficiency, with consistent gains across task horizons. Controlled updates on LIBERO-Long preserve shared prerequisite stages and concentrate improvement at the first stage where rollout outcomes diverge.
