Search papers, labs, and topics across Lattice.
Tournament-GRPO is introduced as a group-wise reward framework for reinforcement learning in open-ended long-form generation, addressing the limitations of pointwise LLM-as-a-judge scoring. It uses multi-round tournaments among same-query rollouts to convert rubric-guided LLM judgments into relative rewards. Experiments on Deep Research Bench demonstrate that Tournament-GRPO outperforms existing reward-design baselines, achieving a 4.52-point overall-score improvement over the strongest baseline, indicating its effectiveness in providing a strong reward signal.
Stop relying on absolute LLM scores for RLHF: relative comparisons via tournaments yield significantly better rewards for long-form generation.
Reinforcement learning in open-ended long-form generation is challenging because reliable reference answers and automatic metrics are often unavailable. Existing rubric-based methods typically rely on pointwise LLM-as-a-judge scoring, but absolute scores are difficult to calibrate across complex responses, may provide weak discrimination among same-query rollouts, and can become saturated during optimization. We propose Tournament-GRPO, a group-wise reward framework that converts rubric-guided LLM judgments into relative rewards through repeated multi-round tournaments among same-query rollouts. Tournament-GRPO compares candidates within groups, accumulates tournament outcomes, and normalizes them into group-wise rewards for GRPO training. Experiments on Deep Research Bench show that Tournament-GRPO consistently outperforms existing reward-design baselines, achieving a 4.52-point overall-score improvement over the strongest baseline. Further analyses show that tournament rewards provide a favorable effectiveness--efficiency trade-off and that tournament design affects training dynamics. These results suggest that rubric-guided tournament comparison provides an effective reward signal for reinforcement learning in open-ended long-form generation.
Tsinghua AI