Search papers, labs, and topics across Lattice.
This paper addresses the instability of standard deviation-based normalization in multi-modal large language models (LLMs) trained with Reinforcement Learning with Verifiable Rewards (RLVR) and Group Relative Policy Optimization (GRPO), which is exacerbated by extreme reward samples. They propose difficulty-aware group normalization (Durian), which regroups samples based on difficulty levels determined by perceptual complexity (visual entropy) and reasoning uncertainty (model confidence) and shares standard deviation within each group. Experiments demonstrate that Durian improves performance on multiple multimodal reasoning benchmarks by preserving intra-group distinctions while mitigating sensitivity to extreme cases.
Multimodal LLMs get a serious reasoning boost from Durian, a difficulty-aware normalization that tames the instability caused by extreme samples and noisy rewards.
Reinforcement Learning with Verifiable Rewards (RLVR) and Group Relative Policy Optimization (GRPO) have significantly advanced the reasoning capabilities of large language models. Extending these methods to multimodal settings, however, faces a critical challenge: the instability of std-based normalization, which is easily distorted by extreme samples with nearly positive or negative rewards. Unlike pure-text LLMs, multimodal models are particularly sensitive to such distortions, as both perceptual and reasoning errors influence their responses. To address this, we characterize each sample by its difficulty, defined through perceptual complexity (measured via visual entropy) and reasoning uncertainty (captured by model confidence). Building on this characterization, we propose difficulty-aware group normalization (Durian), which re-groups samples by difficulty levels and shares the std within each group. Our approach preserves GRPO's intra-group distinctions while eliminating sensitivity to extreme cases, yielding significant performance gains across multiple multimodal reasoning benchmarks.
