Search papers, labs, and topics across Lattice.
This paper addresses the "Vacuous Credit" failure mode in rubric-based reinforcement learning, where reward signals are assigned even when required information is absent. MetaRubric mitigates this by alternating evidence-aware policy optimization with response-guided rubric adaptation, using counterfactual prompts to ensure credit is only given for present evidence. This approach significantly improves performance on medical question-answering benchmarks, outperforming static-judge GRPO by up to 20.40 percentage points.
Rubric-based RL can be misled by "Vacuous Credit," but MetaRubric's adaptive rubric learning achieves substantial accuracy gains on complex QA tasks.
Rubric-based reinforcement learning extends reward-driven optimization to open-ended tasks by assigning partial credit to individual response requirements. However, rubric judges can assign a high criterion score even when the information or action it requires is absent from the response, a failure mode we term Vacuous Credit. Such awards persist after the required information is removed and can reverse the sign of a response's GRPO advantage. To address this problem, we introduce MetaRubric, which alternates evidence-aware policy optimization with response-guided rubric adaptation. We construct counterfactual counterparts by changing one task-relevant fact in each prompt. During policy optimization, credit is assigned only when the response contains sufficient evidence to satisfy the required rubric criterion. After each policy-optimization stage, current policy responses guide revisions to original and counterfactual criteria while preserving the meaning of the original prompt's initial rubric as interpreted under each prompt's facts. We also adapt criterion weights at stage boundaries to better address observed policy errors. Across multiple backbones, MetaRubric improves PubMedQA accuracy by 6.00--20.40 percentage points over static-judge GRPO, with further gains on HealthBench-Hard and two multimodal medical benchmarks.
