Search papers, labs, and topics across Lattice.
SubSearch is introduced as a framework that optimizes LLMs for complex retrieval tasks by providing intrinsic reward signals for intermediate reasoning steps, moving beyond outcome-only supervision. This approach eliminates the need for external supervision by directly optimizing the generator with internally-derived rewards that incentivize high-quality reasoning plans. Experiments across seven QA and multi-hop QA benchmarks demonstrate that rewarding intermediate reasoning steps leads to more robust reasoning traces compared to using only outcome rewards.
Forget expensive human annotations: SubSearch unlocks more robust reasoning in LLMs by directly rewarding intermediate steps with intrinsic rewards, outperforming outcome-only supervision.
Large language models (LLMs) are probabilistic in nature and perform more reliably when augmented with external information. As complex queries often require multi-step reasoning over the retrieved information, with no clear or predetermined reasoning path, they remain challenging. Recent approaches train models using reinforcement learning on the model's outcome, showing promise in improving how models handle complex information. We introduce SubSearch, a specialized framework that shifts from outcome-only supervision to intermediate reward signals that incentivize planning high-quality reasoning. Unlike previous work on process reward modeling, which focuses on training a separate reward model with annotated trajectories by either human annotators or large LLM judges, SubSearch directly optimizes the generator using intrinsic process rewards, which we define as internally-derived rewards, eliminating the need for external supervision, and moving towards autonomous information-intensive reasoning. Experiments on seven benchmarks show that rewarding intermediate reasoning steps with intrinsic rewards leads to more robust reasoning traces in both QA and multi-hop QA datasets over using only outcome rewards. SubSearch can help in building reasoning traces that allow agents to better integrate search engines for complex query answering, while offering a data-efficient alternative to supervised process modeling.
