Search papers, labs, and topics across Lattice.
The paper introduces ShapE-GRPO, a novel reward allocation scheme for training LLMs in multi-candidate generation scenarios, where the goal is to maximize the utility of the entire set of candidates. It addresses the issue of noisy training signals in Group Relative Policy Optimization (GRPO) by using Shapley values to decompose set-level rewards into candidate-specific signals, thus mitigating the "free-riding" problem of poor candidates. Experiments demonstrate that ShapE-GRPO outperforms standard GRPO across various datasets, exhibiting faster convergence.
Stop rewarding all LLM-generated candidates equally: ShapE-GRPO uses Shapley values to fairly distribute credit within sets, leading to better training and faster convergence.
In user-agent interaction scenarios such as recommendation, brainstorming, and code suggestion, Large Language Models (LLMs) often generate sets of candidate recommendations where the objective is to maximize the collective utility of the entire set rather than individual candidates independently. However, existing reinforcement learning post-training paradigms, such as Group Relative Policy Optimization (GRPO), typically assign the same set-level scalar reward to every candidate in the set. This leads to noisy training signals where poor candidates free-ride on the high reward produced by a single strong peer, resulting in suboptimal exploration. To address this, we propose Shapley-Enhanced GRPO (ShapE-GRPO). By leveraging the permutation-invariant nature of set-level utility, we derive a Shapley-enhanced formulation from cooperative game theory to decompose set-level rewards into granular, candidate-specific signals. We show that our formulation preserves the fundamental axioms of the Shapley value while remaining computationally efficient with polynomial-time complexity. Empirically, ShapE-GRPO consistently outperforms standard GRPO across diverse datasets with accelerated convergence during training.
MIT CSAIL