Search papers, labs, and topics across Lattice.
This paper introduces Information Gain-based Rollout Policy Optimization (IGRPO), a novel framework that optimizes the allocation of rollout budgets in multi-turn large language model (LLM) agents by prioritizing the informativeness of intermediate states. By employing a tree-structured rollout approach, IGRPO ensures that more informative branches are expanded more frequently, leading to a more efficient exploration of the decision space. Experimental results across seven challenging search-augmented QA benchmarks show that IGRPO significantly outperforms existing methods while adhering to the same budget constraints, highlighting the effectiveness of its adaptive strategy.
Allocating rollout budgets based on state informativeness allows LLM agents to achieve superior performance in complex decision-making tasks without increasing computational costs.
Reinforcement learning has become a promising paradigm for improving large language model (LLM) agents on long-horizon search tasks, where the agent must make a sequence of intermediate decisions before receiving a final outcome. However, existing methods still face a key limitation: the rollout budget is often allocated without explicitly assessing the utility of intermediate states. As a result, substantial computation may be spent on low-value states, even though different branches can vary drastically in their informativeness. In this paper, we propose Information Gain-based Rollout Policy Optimization (IGRPO), a policy optimization framework that treats intermediate-state informativeness as the organizing principle of rollout collection. Specifically, IGRPO performs budget-aware tree-structured rollouts by allocating expansion budget according to node-level informativeness, so that more informative branches are expanded more frequently while unpromising branches are progressively suppressed. We further demonstrate that the information gain-based rollout induces an explicit limiting teacher distribution over trajectories, which naturally yields a clear policy optimization target, thereby unifying adaptive tree-structured exploration with principled policy learning under a single framework. Experiments on seven challenging search-augmented QA benchmarks demonstrate that IGRPO consistently outperforms strong baselines under the same rollout budget constraints, validating the effectiveness of leveraging the induced teacher distribution to guide policy optimization for long-horizon search agents.
