Search papers, labs, and topics across Lattice.
This paper addresses the supervision-credit gap in outcome-based reinforcement learning by introducing TASPO, a method that converts privileged information (PI) into outcome-grounded action credit. By aggregating PI-induced likelihood shifts at the executable-action level, TASPO provides a more nuanced credit assignment that aligns with verified task outcomes, leading to improved policy optimization. The results show that TASPO outperforms GRPO by 10.6% across three benchmarks and demonstrates better generalization to unseen tasks, highlighting its effectiveness in stabilizing the optimization process.
TASPO bridges the supervision-credit gap in reinforcement learning, leading to a 10.6% performance boost over traditional methods.
Outcome-based reinforcement learning provides verified feedback for language-model agents, but assigns trajectory-level advantage uniformly to all decisions, yielding coarse credit over long-horizon interactions. On-policy self-distillation offers finer supervision by re-evaluating sampled behavior with privileged information (PI) available only during training. However, fine-grained supervision is not necessarily fine-grained credit: PI-induced likelihood changes describe how additional information alters policy preference, but do not directly determine how an executable action should inherit the verified task outcome. This creates a supervision-credit gap. Privileged signals may be irrelevant to the current interaction state, operate at a token granularity misaligned with executable decisions, and lack the outcome semantics required for reinforcement. We introduce TASPO, which converts privileged supervision into outcome-grounded action credit. TASPO constructs decision-applicable PI from verified successful experience, aggregates PI-induced likelihood shifts at the executable-action level, and converts relative action support into positive, bounded, mean-preserving weights on the original trajectory advantage. Thus, the verified outcome determines the update direction and average scale, while PI only redistributes credit across actions. Across three agentic benchmarks, TASPO improves over GRPO by 10.6\% and generalizes better to unseen tasks. Further analysis indicates that TASPO reduces supervision mismatch and that action-level assignment stabilizes the policy optimization process. These findings offer the community another interesting perspective.
