Search papers, labs, and topics across Lattice.
This paper introduces Self-Trained Verification (STV), a method that leverages reference solutions to train a verifier to identify errors in a model's self-generated reasoning steps. STV significantly improves verification-refinement loops at test time, leading to substantial accuracy gains on hard math and scientific reasoning tasks compared to baselines like SFT and RL-based verifiers. Furthermore, Verifier-in-the-Loop training (ViL), which uses the STV verifier's feedback during RL training, boosts the generator's standalone performance by 30%, demonstrating the effectiveness of verification-aware training.
Reference solutions unlock a powerful self-training signal for verifiers, leading to massive gains in reasoning accuracy both during test-time refinement and after training.
Self-improvement at scale has been a longstanding goal for reasoning models, and there are two natural places to do it: at test time, through verification-refinement (V-R) loops; and at training time, through self-training methods. Both are gated by the same bottleneck: the verifier. V-R loops stall when verifier scores inflate while accuracy stagnates, and when feedback is too generic to act on; self-training fails similarly when bad self-generated data are added to training. Better verification would unlock both, but the capability we want to train, i.e., catching self-generated errors, lacks training signal. To address this challenge, we propose self-trained verification (STV). Our key observation is that, while a model cannot catch these errors alone, it can when shown the reference solution. We turn this asymmetry into a supervision target and train the verifier to imitate a more informed version of itself. At test time, STV substantially improves V-R loops on hard problems, while alternatives (e.g., SFT, RL on verifier scores, and even meta-verifiers) do not. STV roughly doubles accuracy on hard math and lifts it 14x on scientific reasoning tasks (1.5% to 21%). At training time, we additionally train the generator using RL with STV verifier's feedback inside the V-R loop - a procedure we call verifier-in-the-loop training (ViL). Starting from an RL-converged generator, ViL yields a further 33% gain in pass@1. More notably, the generator's standalone pass@1, with no verifier at test time, climbs 30% relative past where standard RL had converged. Hence, the next frontier in reasoning on hard problems may lie in how we train for and with verification.
