Search papers, labs, and topics across Lattice.
This paper introduces TRACE, a unified rollout budget allocation framework designed to enhance reward contrast in multi-turn reinforcement learning with verifiable rewards (RLVR). By modeling each turn in a ReAct-style interaction as a distinct node, TRACE enables more effective allocation of rollout resources across both prompt roots and intermediate prefixes, thereby improving the quality of feedback received during policy optimization. The empirical results demonstrate that TRACE achieves significant performance improvements, such as a 2.8-point increase in accuracy on Multi-Hop QA tasks, while maintaining sampling efficiency.
TRACE revolutionizes multi-turn reinforcement learning by transforming how rollout budgets are allocated, leading to enhanced reward contrast and improved agent performance.
Reinforcement learning with verifiable rewards (RLVR) is a promising approach for enhancing reasoning and agentic behavior in large language models. However, rollout-intensive policy optimization is often limited by insufficient reward contrast, arising when overly simple or complex prompts generate low-variance feedback and when outcome-only rewards assign the same terminal assessment to every decision in a multi-turn rollout. Past efforts have focused on allocating available rollout resources to promising prompts, yet they only leverage sample informativeness at the prompt level and neglect variation in prefix-level informativeness across turns within the same rollout. This work targets multi-turn agentic RL by modeling each ReAct-style thought-action-observation turn as a semantically distinct node, allowing budget allocation to extend from prompt roots to turn-level prefixes with further continuations, which naturally forms tree-structured rollouts. We introduce Tree Rollout Allocation for Contrastive Exploration (TRACE), a unified rollout allocation framework that enhances reward contrast within a fixed sampling budget. Technically, TRACE allocates rollout budget to both prompt roots and intermediate prefixes that are most likely to yield mixed terminal rewards. A shared generalizable predictor estimates conditional success probability at these anchors from prefix histories to guide this allocation. The resulting adaptive tree structure enriches outcome-only feedback and amplifies the policy-update signal. Empirically, TRACE achieves competitive performance and efficiency gains on typical agentic benchmarks, e.g., improving Qwen3-14B Multi-Hop QA average accuracy by 2.8 points over competitive baselines at equal sampling cost.
Tsinghua AI