Search papers, labs, and topics across Lattice.
This paper addresses the challenge of catastrophic forgetting in continual reinforcement fine-tuning (RFT) of multimodal large language models, particularly under significant task distributional shifts. The authors introduce Risk-Aware Policy Optimization (RAPO), a dual-channel framework that explicitly governs optimization risk through adaptive sample updates and dynamic risk stratification of training batches. The results show that RAPO significantly reduces forgetting by 79.8% compared to its RLOO backbone while maintaining performance on new tasks, highlighting its effectiveness in continual learning scenarios.
Forgetting can be drastically reduced by nearly 80% in continual learning scenarios without sacrificing performance on new tasks.
Reinforcement fine-tuning (RFT) is widely believed to inherently resist catastrophic forgetting in continual post-training of multimodal large language models. Under pronounced task distributional shifts, however, forgetting across representative RFT algorithms escalates sharply. This stems from the implicit reward-variance regularization inherent to RFT, which proves incapable of suppressing uncontrolled optimization risk. We propose Risk-Aware Policy Optimization (RAPO), the first dual-channel framework for explicit risk governance in continual RFT. On the policy channel, Risk-Aware Policy Scaling adaptively calibrates per-sample update magnitude via rollout reliability and Fisher-inspired local predictive sensitivity; on the data channel, Risk-Aware Dynamic Bucket Sampling reorganizes training batches through dynamic risk stratification, steering optimization toward informative yet stable samples. As a plug-and-play strategy requiring no cross-task memory, RAPO generalizes to any RFT algorithm without modification. On the public MLLM-CL benchmark, RAPO reduces final forgetting by 79.8% relative to its RLOO backbone while retaining new-task competitiveness.
