Search papers, labs, and topics across Lattice.
This paper introduces Medical Reasoning-aware Policy Optimization (MRPO), a reinforcement learning algorithm designed to enhance multimodal large language models in medical visual question answering (VQA) by addressing cascading errors from early-stage reasoning failures. By assigning exponentially larger penalties to earlier invalid reasoning steps when the final answer is incorrect, MRPO effectively breaks failure cascades and improves both reasoning quality and accuracy. The results demonstrate that MRPO significantly outperforms standard approaches and even surpasses larger medical models, reducing early-stage reasoning failures from 64.0% to 13.0% and achieving a notable performance increase of 2.79 points on specific benchmarks.
Early-stage reasoning failures can be drastically reduced from 64% to 13% with a novel RL approach that penalizes cascading errors in medical VQA.
Recent multimodal large language models have shown great promise in clinical image reasoning, but existing post-training pipelines remain predominantly outcome-centric, relying on final answer correctness or sequence-level preferences. This suffers from sparse credit assignment, making it difficult to optimize the reasoning process essential for clinical applications. Our analysis reveals that cascading errors from early-stage reasoning failures are a leading cause of incorrect predictions in medical visual question answering (VQA) benchmarks. Motivated by this, we propose Medical Reasoning-aware Policy Optimization (MRPO), an RL algorithm that incorporates step-wise process rewards. When the final answer is incorrect, MRPO assigns exponentially larger penalties to tokens in earlier invalid reasoning steps, breaking failure cascades without compromising successful paths. Across three multimodal LLM backbones, MRPO consistently outperforms standard GRPO and a recent RL baseline, and on Qwen3-VL-8B-Instruct even surpasses substantially larger medical MLLMs such as HuatuoGPT-Vision-34B by 2.79 points. Moreover, MRPO reduces early-stage reasoning failures from 64.0% to 13.0%, showing that targeted mitigation of cascading failures improves both reasoning quality and final answer accuracy. Our code is available at https://github.com/dmis-lab/MRPO
