Track the frontier, from condensed matter to cosmology.
Search papers, labs, and topics across Lattice.
This paper addresses the challenge of multi-reward reinforcement learning for language models by proposing MRTPO, a method for reward-specific token-level credit assignment. Unlike prior work that distributes reward advantages uniformly across a response, MRTPO leverages verifier information to isolate and assign each reward's contribution to its relevant tokens. Empirical results on text-to-SQL, multi-constraint instruction following, and tool calling demonstrate MRTPO's faster convergence and superior performance.
Assigning reward credit to specific tokens, rather than uniformly, dramatically improves multi-reward RL for LLMs.
Language models are increasingly trained against several verifiable rewards at once, such as the correctness and the format of a tool call, or the individual constraints of a complex instruction. Reward-decoupled normalization (GDPO) normalizes each reward separately so that different reward combinations keep distinct advantages, but it still spreads every reward's advantage evenly over the whole response, although each verifier checks only a few positions. We propose MRTPO, which assigns the advantage of each reward to the tokens relevant to it, using the information its verifier checks, without auxiliary models or token-level annotations. For each reward, the policy replays its own response with and without that reward's verifier information, and the per-token Jensen-Shannon distance between the two next-token distributions measures each token's relevance and reassigns the advantage. On text-to-SQL, multi-constraint instruction following and tool calling, MRTPO converges faster and achieves better performance than the compared methods.