Search papers, labs, and topics across Lattice.
This paper introduces Parallel Shapley, a novel reinforcement learning framework that utilizes Shapley values to attribute fine-grained contributions of individual reasoning paths in large language models (LLMs) during multi-step reasoning tasks. By treating each reasoning path as a player in a cooperative game, the framework addresses the challenge of uniform reward distribution, which can obscure learning signals and destabilize training. Experimental results demonstrate that Parallel Shapley not only surpasses existing baselines in performance on mathematical reasoning benchmarks but also enhances the stability and interpretability of the training process.
"Parallel Shapley effectively 'fishes out the free riders' in reasoning paths, leading to more stable and interpretable training outcomes for LLMs."
Large Language Models (LLMs) excel at multi-step reasoning, yet current parallel reasoning approaches often fail to distinguish the contributions of individual reasoning paths. Many paths may be redundant, misleading, or even detrimental, but outcome-level rewards assign uniform reward, leading to ambiguous learning signals and unstable training. We propose Parallel Shapley, a reinforcement learning framework that attributes fine-grained, path-level contributions in multi-path reasoning. Treating each path as a player in a cooperative game, we leverage Shapley values to quantify marginal contributions, using a generative reward model to evaluate path utilities and Monte Carlo sampling for efficient approximation. Experiments on mathematical reasoning benchmarks show that Parallel Shapley outperforms existing baselines while providing more stable and interpretable training. Our framework effectively "fishes out the free riders," assigning reward proportionally and improving multi-path reasoning in LLMs.
