Search papers, labs, and topics across Lattice.
This paper introduces Gaussian Importance sampling Policy Optimization (GIPO), a novel policy optimization objective for reinforcement learning that uses a Gaussian trust weight based on log-importance ratios to softly dampen extreme importance ratios. GIPO provides an implicit, tunable constraint on update magnitude and demonstrates robustness under finite-sample estimation. Experiments across various replay buffer sizes show GIPO achieves state-of-the-art performance compared to clipping-based baselines, exhibiting improved sample efficiency, bias-variance trade-off, and training stability.
Ditch hard clipping: GIPO's Gaussian-weighted importance sampling offers a smoother, more stable RL policy optimization, especially when dealing with stale or limited data.
Post-training with reinforcement learning (RL) has recently shown strong promise for advancing multimodal agents beyond supervised imitation. However, RL remains limited by poor data efficiency, particularly in settings where interaction data are scarce and quickly become outdated. To address this challenge, GIPO (Gaussian Importance sampling Policy Optimization) is proposed as a policy optimization objective based on truncated importance sampling, replacing hard clipping with a log-ratio-based Gaussian trust weight to softly damp extreme importance ratios while maintaining non-zero gradients. Theoretical analysis shows that GIPO introduces an implicit, tunable constraint on the update magnitude, while concentration bounds guarantee robustness and stability under finite-sample estimation. Experimental results show that GIPO achieves state-of-the-art performance among clipping-based baselines across a wide range of replay buffer sizes, from near on-policy to highly stale data, while exhibiting superior bias--variance trade-off, high training stability and improved sample efficiency.
Tsinghua AI