Search papers, labs, and topics across Lattice.
The paper introduces HiPER, a hierarchical reinforcement learning framework for training LLM agents in long-horizon tasks with sparse rewards, by separating high-level planning (subgoal generation) from low-level execution. HiPER addresses the credit assignment problem by using hierarchical advantage estimation (HAE), which provides unbiased gradient estimates and reduces variance compared to flat advantage estimation. Empirical results on ALFWorld and WebShop benchmarks demonstrate that HiPER achieves state-of-the-art performance, particularly on long-horizon tasks, highlighting the benefits of explicit hierarchical decomposition.
LLM agents can learn complex, multi-turn tasks far more effectively by explicitly separating planning from execution, using a hierarchical RL approach with carefully designed credit assignment.
Training LLMs as interactive agents for multi-turn decision-making remains challenging, particularly in long-horizon tasks with sparse and delayed rewards, where agents must execute extended sequences of actions before receiving meaningful feedback. Most existing reinforcement learning (RL) approaches model LLM agents as flat policies operating at a single time scale, selecting one action at each turn. In sparse-reward settings, such flat policies must propagate credit across the entire trajectory without explicit temporal abstraction, which often leads to unstable optimization and inefficient credit assignment. We propose HiPER, a novel Hierarchical Plan-Execute RL framework that explicitly separates high-level planning from low-level execution. HiPER factorizes the policy into a high-level planner that proposes subgoals and a low-level executor that carries them out over multiple action steps. To align optimization with this structure, we introduce a key technique called hierarchical advantage estimation (HAE), which carefully assigns credit at both the planning and execution levels. By aggregating returns over the execution of each subgoal and coordinating updates across the two levels, HAE provides an unbiased gradient estimator and provably reduces variance compared to flat generalized advantage estimation. Empirically, HiPER achieves state-of-the-art performance on challenging interactive benchmarks, reaching 97.4\% success on ALFWorld and 83.3\% on WebShop with Qwen2.5-7B-Instruct (+6.6\% and +8.3\% over the best prior method), with especially large gains on long-horizon tasks requiring multiple dependent subtasks. These results highlight the importance of explicit hierarchical decomposition for scalable RL training of multi-turn LLM agents.
