Search papers, labs, and topics across Lattice.
This paper introduces HindSearch, a novel hindsight self-distillation method for search-augmented reinforcement learning (RL) that critiques failed trajectories to enhance learning. By employing a frozen judge to generate critiques based on the gold answer, the approach provides valuable auxiliary signals that improve the agent's search actions. The results demonstrate a significant performance boost, achieving a 39.4% average exact match (EM) on a standard benchmark suite, highlighting the importance of leveraging hindsight in RL training.
Hindsight critiques can transform failed trajectories into powerful learning signals, boosting search-augmented RL performance by nearly 40%.
Search-augmented LM agents are typically trained with a binary exact-match reward, which throws away most of what a failed trajectory tells us about why it failed. We introduce HindSearch, a hindsight self-distillation procedure for GRPO: after each rollout, a frozen judge writes a short critique of every failed trajectory using the gold answer, and the critique supplies an auxiliary on-policy distillation signal on the student's search actions. On the standard seven-benchmark suite with Qwen2.5-3B-Instruct, HindSearch reaches 39.4% average EM, outperforming prior search-RL baselines. Removing the judge's access to the gold answer erases most of the gain, isolating hindsight as the source of the improvement.
