Search papers, labs, and topics across Lattice.
This study investigates the phenomenon of rank inversion in reinforcement learning from human feedback (RLHF) due to overtraining in supervised fine-tuning (SFT), specifically analyzing how SFT depth affects performance in models Qwen2.5-Coder-3B and DeepSeek-Coder-6.7B. The authors find that while increasing SFT depth improves pre-RL performance, it can lead to significant drops in GRPO outcomes, particularly when the rollout distribution becomes overly compressed. A novel two-stage diagnostic approach is proposed, which utilizes pre-RL entropy and early GRPO entropy monitoring to identify and mitigate high-risk checkpoints, thereby enhancing training efficiency and effectiveness.
Overtraining in SFT can lead to rank inversion, with GRPO performance plummeting despite improved pre-RL metrics, highlighting a critical failure mode in RLHF training.
The standard heuristic of selecting the SFT checkpoint with the highest pass@1 for GRPO can fail when SFT compresses the rollout distribution. For binary rewards, the expected within group advantage variance is $p(1{-}p)(g{-}1)/g$; when early GRPO drives $p$ below $p^*(g)$, most groups have identical rewards and provide no group relative signal. We study SFT depth ladders for Qwen2.5-Coder-3B and DeepSeek-Coder-6.7B. We test Qwen2.5-Coder-3B across five depths and three seeds, and DeepSeek-Coder-6.7B across four matched depths and three seeds. On Qwen, pre RL pass@1 rises with SFT depth, but peak GRPO pass@10 falls from $0.806$ to $0.481$ (3 seed mean, $n{=}20$); pre RL entropy is positively associated with the GRPO outcome ($蟻{=}{+}0.69$). On DeepSeek, pass@1 remains far above $p^*(8){=}0.083$, and GRPO outcomes compress rather than invert. A two stage diagnostic, combining pre RL entropy triage with an early GRPO entropy monitor, flags high risk checkpoints and can stop failing runs early. Simple KL to reference regularisation and label smoothing variants do not rescue the collapsed Qwen checkpoint in our setting, suggesting the failure is not a trivial GRPO hyperparameter artefact.
