Search papers, labs, and topics across Lattice.
This paper introduces Selective Trajectory-Aware Policy Optimization (STAPO), a novel hierarchical reinforcement learning framework designed to address trajectory neglect in LLM agents during long-horizon tasks. By employing normalized entropy to identify outlier steps linked to neglect, STAPO optimizes these steps through a combination of trajectory-aware rewards and penalties, significantly improving the agents' focus on task goals. Experimental results across multiple benchmarks, including ALFWorld and WebShop, show that STAPO not only achieves state-of-the-art performance but also effectively mitigates the issues associated with trajectory neglect.
Trajectory neglect in LLM agents can be significantly reduced using a novel reward mechanism that enhances focus on task goals without sacrificing training stability.
Reinforcement Learning (RL) is the dominant paradigm for training Large Language Model (LLM) agents on long-horizon tasks. However, sparse and delayed rewards often lead to trajectory neglect, in which agents lose focus on the task goal and interaction history at intermediate steps. Prior work has explored step-level supervision using Shannon-entropy-based uncertainty signals, which conflate inherent state complexity with agent confidence and therefore provide unreliable estimates of decision reliability. To address this issue, we propose normalized entropy, which measures confidence deviations relative to an agent's average behavior under a given state, thereby strengthening the association between low-quality actions and trajectory neglect. Building on this insight, we introduce Selective Trajectory-Aware Policy Optimization (STAPO), a hierarchical group-based RL framework. STAPO leverages normalized entropy to locate outlier steps associated with trajectory neglect and optimizes them via a joint mechanism of trajectory-aware reward and trajectory-independent penalty, enhancing trajectory awareness while preserving training stability. Extensive experiments on ALFWorld, WebShop, and Search-Augmented QA demonstrate that STAPO achieves state-of-the-art performance while substantially alleviating trajectory neglect, validating its effectiveness and robustness for agentic tasks.
