Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
RM-OPD addresses reward sparsity and cross-domain interference in Mix-RL for Generative Reward Models by separating specialist acquisition and capability integration. The framework first trains domain-specialized teacher models via single-domain GRPO, then distills them into a student model using routed on-policy distillation with domain contribution reweighting and optional criteria context distillation. This two-stage approach consistently outperforms Mix-RL and other baselines, demonstrating superior performance and an emergent teacher-surpassing effect.
A novel two-stage distillation framework achieves superior generalist reward model performance by first cultivating domain specialists and then integrating them, even surpassing individual teacher capabilities.
Reinforcement Learning on Mixed-domain preference data (Mix-RL) has become the mainstream approach for training general-purpose Generative Reward Models (GRMs). However, Mix-RL suffers from two critical bottlenecks: the reward sparsity induced by outcome rewards, and the cross-domain interference arising from jointly optimizing heterogeneous capabilities, which together give rise to imbalanced capability acquisition and pervasive reward hacking. Inspired by the success of On-Policy Distillation (OPD) in multi-domain capability integration, we introduce RM-OPD (Reward Modeling via On-Policy Distillation), a two-stage framework that separates specialist acquisition from capability integration. First, we cultivate domain-specialized teacher models via single-domain GRPO fine-tuning, allowing each expert to reach its performance ceiling in isolation; then we consolidate these teachers into one student model through (i) routed on-policy distillation, which provides dense signed token-level supervision on student-generated critiques; (ii) domain contribution reweighting, which reweights token guidance toward target domain shares, preventing specific domains from dominating optimization; and (iii) optional criteria context distillation, which conditions teachers on domain-specific evaluation criteria to sharpen token-level supervision. Across multiple RM benchmarks and two base models, RM-OPD consistently outperforms vanilla Mix-RL and other strong baselines, achieving overall superior performance and exhibiting an emergent "teacher-surpassing" effect. These results establish RM-OPD as a promising paradigm for building generalist GRMs.