Search papers, labs, and topics across Lattice.
This paper introduces TAPO, a novel framework for Reinforcement Learning (RL) that enhances policy optimization for Large Language Model (LLM) agents by incorporating transition supervision from environmental feedback. By alternating between policy optimization and action-conditioned next-observation prediction, TAPO effectively leverages dense supervisory signals to improve the model's understanding of environmental dynamics. Experimental results on WebShop and ALFWorld show that TAPO significantly outperforms traditional policy optimization methods, highlighting its effectiveness as a lightweight enhancement for existing RL algorithms.
Transition supervision can dramatically boost LLM agent performance, outperforming standard policy optimization by leveraging environmental feedback.
Recently, Reinforcement Learning (RL) has emerged as a crucial paradigm for the post-training of Large Language Model (LLM) agents. However, existing methods predominantly rely on sparse task rewards for policy optimization, failing to fully exploit another class of inherently dense supervisory signals naturally present during online interaction: environmental feedback following action execution. Recent theoretical studies suggest that generalization in multi-step, goal-oriented tasks hinges on predictive knowledge of environmental consequences. Inspired by this, we propose TAPO: Transition-Aware Policy Optimization for LLM Agents, a unified training framework that alternates between policy optimization and transition supervision. Beyond standard RL updates, TAPO repurposes rollout data to apply action-conditioned next-observation prediction supervision on a shared backbone model. This approach enhances the model's sensitivity to environmental transition dynamics and action consequences while concurrently optimizing the policy. It serves as a computationally lightweight, plug-and-play enhancement module for existing agent RL algorithms, requiring no additional expert data, extra sampling costs, or inference-time overhead. We conduct systematic experiments on WebShop and ALFWorld, integrating foundation models of various scales with different policy optimization algorithms. Empirical results demonstrate that TAPO consistently improves task performance over pure policy optimization baselines.
