Search papers, labs, and topics across Lattice.
This paper investigates the issue of reward hacking in reinforcement learning (RL) systems trained with rubric-based rewards, demonstrating that policies can exploit fixed rubrics over time. By applying Group Relative Policy Optimization (GRPO) to the Qwen3-8B model on medical and science rubrics, the authors observe a significant divergence between scores from the training judge and a stronger gold judge, indicating that the model is indeed engaging in reward hacking. To address this, they introduce "Rubric Dropout," a method that randomly omits parts of the rubric during training, which effectively mitigates reward hacking without sacrificing performance on out-of-distribution benchmarks.
Reward hacking can be mitigated with a simple one-line fix that improves out-of-distribution performance while keeping training robust.
Reinforcement learning against rubrics, lists of criteria graded by an LLM judge, has become a standard way to post-train language models on tasks with no deterministic answer. The rubric, however, is a fixed proxy for quality, never a complete description of it, and a policy trained against it long enough will learn to exploit the difference. We measure this directly. Training Qwen3-8B with Group Relative Policy Optimization (GRPO) on medical and science rubrics and grading out-of-distribution (OOD) benchmarks with both the training judge and a stronger gold judge, we find that the two scores diverge during training. The training judge's score keeps climbing while the gold judge's score peaks and then falls, by 3 points on HealthBench-Hard and by 22 points on ResearchQA. A judge with a fixed bias would shift the gold curve by a constant, not send it down while the training score rises, so the divergence is reward hacking, not judge noise. We propose Rubric Dropout, a one-line fix borrowed from neuron dropout. At every step, we randomly drop a subset of the rubric's criteria before computing the reward, so the policy never optimizes the same rubric twice. The dropped subset is shared across each rollout group, so GRPO's group-relative advantages stay comparable, and evaluation always uses the full rubric. Comparing no dropout against dropout at 30% and 50% on both benchmark pairs, dropout raises the OOD gold score at every matched checkpoint (+1 to +2 points on HealthBench-Hard, +6 to +7 points on ResearchQA), lowers the two hacking measures we track, and costs nothing in domain. Sweeping the dropout fraction shows a broad 30-50% sweet spot, while the natural alternative, reweighting criteria by how useful they are to training, performs worse than no intervention at all in our setting.
