Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper addresses the credit assignment problem in training search agents for LLMs by introducing intermediate supervision beyond sparse, outcome-based rewards. It systematically investigates various reward-shaping and credit-assignment strategies for retrieval steps and develops a framework combining these with final outcome rewards. Experiments show that this approach significantly improves aggregate search-agent performance and highlights the critical impact of intermediate signal choice and credit assignment location on training dynamics.
Intermediate retrieval signals, not just final outcomes, unlock efficient credit assignment for training LLM search agents.
Search agents enable Large Language Models (LLMs) to iteratively retrieve and use information for complex multi-hop questions. Reinforcement Learning with Verifiable Rewards (RLVR) offers a promising approach for post-training such agents, but its reliance on sparse, outcome-based supervision can make credit assignment difficult and limit learning efficiency. In this paper, we systematically investigate how intermediate supervision can improve reinforcement learning for search agents. We study a range of reward-shaping and credit-assignment strategies that provide learning signals from intermediate retrieval steps. Building on these insights, we develop a training framework that combines intermediate signals with final outcome rewards to improve learning from multi-step search trajectories. Experiments across multiple benchmarks under matched training conditions demonstrate improvements in aggregate search-agent performance and show that both the choice of intermediate signal and where its credit is assigned affect training behaviour. These findings show that reward design and credit assignment are important design dimensions for training effective search agents.