Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces a dense process supervision method for reinforcement learning (RL) search agents that utilizes fact utility estimation to improve credit assignment during training. By extracting structured facts from observations and organizing them into a fact store, the authors cluster semantically equivalent facts and apply Bayesian estimation to infer their utility, converting these estimates into dense rewards for RL training. Experimental results demonstrate that this approach significantly enhances performance on both single-hop and multi-hop question-answering benchmarks compared to traditional outcome reward methods.
Dense process supervision can transform RL training by providing step-level rewards that enhance credit assignment, leading to superior performance in complex reasoning tasks.
Reinforcement learning (RL) for search agents typically relies on outcome rewards. However, it often fails to achieve effective credit assignment, due to the unclear value of intermediate steps. It is hard to separate their contributions from the final result. In this paper, we propose a dense process supervision method based on fact utility estimation, which models the reasoning process as the accumulation of discrete evidence facts. We first extract structured facts from raw observations and organize them into an explicit fact store. To support credit assignment, we then cluster semantically equivalent facts and infer the posterior utility of each fact cluster using Bayesian estimation over group rollouts. Finally, we convert the estimated fact utilities into dense step-level rewards to guide RL training. Experiments on seven single-hop and multi-hop QA benchmarks show that our method consistently outperforms existing baselines. Ablation studies validate clear relative improvements on multi-hop QA compared to outcome reward-only training.