Search papers, labs, and topics across Lattice.
This paper introduces GFlowRL, a novel reinforcement learning algorithm that scales GFlowNet methods to large language models by eliminating the need for a learned partition function, which previously caused gradient instability. By utilizing an in-batch Monte Carlo estimate for reward distribution matching, GFlowRL achieves superior performance on math, code, and adversarial benchmarks, outperforming existing methods like FlowRL. The results demonstrate that GFlowRL can stably scale across various model architectures, achieving high ratings and robust performance even in challenging scenarios.
GFlowRL achieves unprecedented stability and performance in large language models by eliminating the problematic learned partition function, setting a new standard for GFlowNet-style reinforcement learning.
Generative Flow Networks (GFlowNets) offer a promising alternative to reward-maximizing reinforcement learning (RL) for large reasoning models, encouraging diverse reasoning paths by matching reward distributions rather than collapsing to dominant modes. Recent work shows promise on math and code, but scaling GFlowNet-style RL to modern post-training pipelines remains difficult: as model size, rollout horizon, reward noise, and distributed-systems complexity grow together, a learned prompt-conditional partition function becomes a source of gradient instability and engineering overhead rather than a useful normalizer. Through systematic analysis, we find that the learned partition function, previously treated as essential, can be replaced by an in-batch Monte Carlo estimate computed from the rollout group already required for training. We propose GFlowRL, a streamlined GFlowNet-style RL algorithm that removes the auxiliary partition network entirely while preserving the reward-distribution-matching objective, completed by two stabilizers: importance-sampling correction for rollout/trainer drift and asymmetric flow-gap clipping for outlier residuals. GFlowRL exceeds all counterparts on math, code, and adversarial red-teaming benchmarks, reaching a Codeforces rating of 2048 at the 14B scale (within 25 Elo of o3-mini) and attaining the highest average ASR@1 on AdvBench and HarmBench, outperforming the previous SOTA multi-turn attacker in a regime where FlowRL, a prior GFlowNet-style method, diverges. The same recipe transfers to all evaluated MoE configurations up to 235B parameters, where FlowRL again fails to converge. To our knowledge, GFlowRL is the first GFlowNet-style RL algorithm to scale stably across both dense and sparse architectures. Code will be at: https://github.com/microsoft/gflowrl
Microsoft Research