Search papers, labs, and topics across Lattice.
EvoRubric introduces a co-evolutionary RL framework for open-ended generation, unifying response and rubric generation under a single policy that dynamically alternates between a Reasoner and a Rubric Generator. To ensure reliable reward signals, it incorporates a multi-level verification pipeline with a meta-verifier, zero-variance pruning, and Leave-One-Out peer consensus. Experiments across diverse domains demonstrate that EvoRubric outperforms static and external-LLM-driven alignment methods, and can even improve upon expert-annotated rubrics by discovering novel discriminative dimensions.
Forget static rubrics and expensive external models: EvoRubric co-evolves a single policy to generate both responses and the rubrics to evaluate them, outperforming traditional RLHF methods in open-ended generation tasks.
Reinforcement Learning (RL) has significantly advanced Large Language Models (LLMs) in verifiable domains, but aligning models for open-ended generation remains profoundly challenging due to the lack of definitive rewards. Current rubric-based RL methods mitigate this by employing explicit criteria; however, they rely heavily on static, human-annotated rubrics that inevitably cause policy lag, or expensive external proprietary models for dynamic updates. In this paper, we propose EvoRubric, a novel single-policy co-evolutionary RL framework that eliminates the reliance on static criteria and on external rubric generators. By unifying response generation and rubric generation under a single parameterized policy, EvoRubric dynamically alternates between a Reasoner and a Rubric Generator. To prevent reward hacking and ensure the reliability of generated signals, we introduce a multi-level verification pipeline featuring a meta-verifier, zero-variance pruning, and a Leave-One-Out peer consensus mechanism. Validated criteria are dynamically archived into a memory pool, yielding dense, multi-objective rewards to continuously co-optimize both roles. Extensive experiments across Medical, Writing, and Science domains demonstrate that EvoRubric consistently outperforms traditional static and external-LLM-driven alignment methods. Notably, our framework is compatible with human-expert priors. When initialized with expert-annotated rubrics, EvoRubric can further uncover novel, discriminative dimensions, achieving better performance than relying solely on static expert annotations.
DAMO