Search papers, labs, and topics across Lattice.
This paper introduces EvoRubrics, a co-evolutionary reinforcement learning framework that enables a Policy LLM and a Rubric Generator to dynamically adapt their criteria through adversarial interaction during training. By allowing the rubric to evolve alongside the policy, EvoRubrics addresses the issue of reward saturation and enhances the discriminative power of evaluation metrics in open-ended tasks. Experimental results demonstrate that EvoRubrics outperforms both static and existing dynamic rubric methods, with a self-supervised variant achieving significant improvements without external supervision.
Dynamic rubrics that evolve with the policy can significantly enhance reinforcement learning performance, even without external supervision.
Rubric-based rewards offer interpretable and fine-grained optimization signals for reinforcement learning in open-ended tasks where verifiable answers are unavailable. However, pre-constructed rubrics remain static throughout training, creating a fundamental mismatch with the evolving policy: fixed criteria gradually lose discriminative power as the model improves, leading to reward saturation and potential hacking. Recent dynamic rubric methods partially address this but rely on external frontier models or ground-truth answers, and update rubrics only at coarse granularity. We propose EvoRubrics, a co-evolutionary RL framework where a Policy LLM and a Rubric Generator jointly improve through adversarial interaction within each training step. As the policy improves under the rubric generator's guidance, the rubric generator adapts its criteria to remain discriminative and informative, enabling evaluation to track the policy in real time and naturally inducing an automatic curriculum. Experiments show that EvoRubrics consistently outperforms static and dynamic rubric baselines across benchmarks. The learned Rubric Generator further generalizes as a transferable reward model. Notably, even a fully self-supervised variant without any external supervision achieves meaningful gains, suggesting that co-evolution between generation and evaluation alone can provide sufficiently rich learning signals. Our code is publicly available at https://anonymous.4open.science/r/EvoRubrics-2155/.
