Search papers, labs, and topics across Lattice.
The paper addresses the issue of reasoning-answer inconsistency in Reinforcement Learning with Verifiable Rewards (RLVR) for multimodal reasoning, where correct answers are achieved through flawed reasoning. They compare reward models (RMs) and Generative Rewards (GRs) for trajectory supervision, finding limitations in both. To address these limitations, they introduce Groupwise Ranking Reward, which ranks verifier-passed trajectories for the same prompt to redistribute reward, leading to improved reliability-conditioned accuracy.
Rewarding *correct* answers in multimodal reasoning can actually *worsen* reasoning quality, but a simple groupwise ranking of solution trajectories significantly boosts reliability.
Reinforcement Learning with Verifiable Rewards (RLVR) improves multimodal reasoning by rewarding verifiable final answers. Yet answer-correct trajectories may still rely on incomplete derivations, weak evidence, or statements that contradict their conclusions. This gap between answer correctness and reasoning validity, which we call reasoning-answer inconsistency, motivates trajectory supervision in multimodal RL. We compare two main approaches: reward models (RMs), and Generative Rewards (GRs). RMs are efficient and help early in training, but their gains weaken as the policy distribution shifts; GRs improve performance, but may give unstable rewards and computationally expensive. We therefore propose Groupwise Ranking Reward, which ranks verifier-passed trajectories for the same prompt in one pass and redistributes reward accordingly. Groupwise comparison better separates stronger and weaker correct trajectories with lower judge overhead than GRs. Experiments show that RLVR aggravates reasoning-answer inconsistency, while trajectory supervision alleviates it. Groupwise Ranking Reward performs best overall, improving reliability-conditioned accuracy from 47.4% to 54.7% over RLVR.
