Search papers, labs, and topics across Lattice.
This paper addresses the problem of distribution sharpening in math RLVR, where models trained with verifiable rewards exhibit improved pass@1 accuracy but reduced solution coverage. They introduce Distribution-Aligned Hint Synthesis (DAHS) to generate teacher hints conditioned on student-style responses, mitigating teacher-student distribution mismatch. They further propose Backward Hint Annealing (BHA) to reduce hint exposure during training, preserving no-hint updates. Experiments on AIME benchmarks using Qwen3-1.7B-Base and Llama-3.2-1B-Instruct demonstrate improved pass@1 and pass@2048 accuracy, particularly in the large-$k$ regime, showing the effectiveness of hint scaffolding.
Scaffolding LLMs with hints during RL training can boost both initial accuracy *and* long-horizon reasoning performance, but only if the hints mimic student behavior and are gradually withdrawn.
Reinforcement learning with verifiable rewards (RLVR) can improve low-$k$ reasoning accuracy while narrowing solution coverage on challenging math questions, and pass@1 gains do not necessarily translate into better large-$k$ performance. Existing hint-based approaches can make challenging questions trainable, but they leave two issues underexplored: teacher-student distribution mismatch and the need to reduce hint exposure to match no-hint evaluation. We address these issues through two components. Distribution-Aligned Hint Synthesis (DAHS) constructs verified teacher hints conditioned on student-style responses. Backward Hint Annealing (BHA) anneals hint exposure across difficulty buckets and uses per-question hint dropout to preserve no-hint updates throughout RL training. We evaluate the method in math RLVR under the DAPO training framework across AIME24, AIME25, and AIME26 using $\texttt{Qwen3-1.7B-Base}$ and $\texttt{Llama-3.2-1B-Instruct}$. On $\texttt{Qwen3-1.7B-Base}$, our method improves both pass@1 and pass@2048 relative to DAPO across the three AIME benchmarks. On $\texttt{Llama-3.2-1B-Instruct}$, the gains are concentrated in the large-$k$ regime. These results suggest that, in math RLVR, hint scaffolding is effective when it restores learnable updates on challenging questions early in training and is then gradually removed before no-hint evaluation.
