Search papers, labs, and topics across Lattice.
This paper introduces ScoRe-Flow, a reinforcement learning fine-tuning method for Flow Matching (FM) policies that leverages score function modulation to improve exploration and stability. By combining drift modulation with learned variance prediction, ScoRe-Flow achieves decoupled control over the mean and variance of stochastic transitions. Experiments on D4RL, Robomimic, and Franka Kitchen benchmarks demonstrate faster convergence and higher success rates compared to state-of-the-art flow-based RL methods.
Score function modulation unlocks 2.4x faster RL fine-tuning of Flow Matching policies, outperforming prior methods by decoupling control over the mean and variance of stochastic transitions.
Flow Matching (FM) policies have emerged as an efficient backbone for robotic control, offering fast and expressive action generation that underpins recent large-scale embodied AI systems. However, FM policies trained via imitation learning inherit the limitations of demonstration data; surpassing suboptimal behaviors requires reinforcement learning (RL) fine-tuning. Recent methods convert deterministic flows into stochastic differential equations (SDEs) with learnable noise injection, enabling exploration and tractable likelihoods, but such noise-only control can compromise training efficiency when demonstrations already provide strong priors. We observe that modulating the drift via the score function, i.e., the gradient of log-density, steers exploration toward high-probability regions, improving stability. The score admits a closed-form expression from the velocity field, requiring no auxiliary networks. Based on this, we propose ScoRe-Flow, a score-based RL fine-tuning method that combines drift modulation with learned variance prediction to achieve decoupled control over the mean and variance of stochastic transitions. Experiments demonstrate that ScoRe-Flow achieves 2.4x faster convergence than flow-based SOTA on D4RL locomotion tasks and up to 5.4% higher success rates on Robomimic and Franka Kitchen manipulation tasks.
