Search papers, labs, and topics across Lattice.
This paper introduces BiCAA, a bidirectional credit assignment framework that enhances multi-step search agents by providing dense, stepwise supervision signals during the reasoning process. By integrating forward solvability gain and hindsight success criticality, BiCAA effectively mitigates the training instability and redundancy issues associated with traditional outcome-only supervision methods. Experimental results demonstrate that BiCAA not only stabilizes policy optimization but also improves performance on search-augmented question answering benchmarks.
Sparse supervision in multi-step search tasks leads to instability, but BiCAA's bidirectional credit assignment transforms how agents learn from each reasoning step.
Multi-step search is a fundamental capability for search agents, enabling them to iteratively acquire, refine, and integrate external evidence for complex reasoning QA. However, vanilla GRPO allocates rewards exclusively based on the model's final outputs, yielding outcome-only supervision with no supervisory signals for intermediate reasoning steps. Such sparse supervision easily causes training instability and redundant search behaviors on multi-step search tasks. To mitigate this limitation, we adopt process reward to deliver stepwise supervision signals. For this process reward, we propose two complementary criteria to judge each search step: whether the step yields new evidence to facilitate problem solving, and whether it forms an efficient, pivotal intermediate decision within the overall reasoning trajectory. Building on this insight, we propose BiCAA: a bidirectional credit assignment framework that delivers dense, distinguishing process rewards for search-augmented agents. BiCAA builds bidirectional process rewards by fusing two complementary signals: forward solvability gain and hindsight success criticality. The former quantifies step-wise improvements in answer plausibility, while the latter evaluates each step's necessity for final success via hindsight outcome-based criticality scoring. We modulate and aggregate the two signals and then fuse them with the outcome reward. Experiments on search-augmented QA benchmarks show that BiCAA stabilizes policy optimization, reduces redundant search behavior, and achieves competitive performance.
