Search papers, labs, and topics across Lattice.
This paper introduces DigenRL, a disaggregated reinforcement learning framework specifically designed for diffusion-based generative large language models (LLMs), addressing the inefficiencies of existing colocated implementations. By employing generation-axis pipeline (GAP) and time-step parallelism (TSP), along with an elastic trainer-assisted generation (TAG) approach, DigenRL enables flexible resource allocation and efficient task scheduling across heterogeneous GPUs. Experimental results demonstrate that DigenRL achieves significant throughput improvements of 1.56 to 2.10 times over leading diffusion RL systems, highlighting its potential for optimizing resource utilization in generative tasks.
DigenRL achieves up to 2.10x throughput gains for diffusion-based generative LLMs by disaggregating resource allocation and optimizing execution pipelines.
Reinforcement learning (RL) has become a dominant post-training paradigm, driving the emergence of high-performance RL systems such as veRL for autoregressive large language models (LLMs). In parallel, diffusion-oriented RL algorithms, e.g., DanceGRPO and FlowGRPO, have rapidly expanded the scope of RL from language reasoning to diffusion-based visual and flow-based generation. However, efficient RL systems for diffusion generative LLMs remain underexplored. Existing implementations, e.g., veRL-Omni, still rely on colocated execution, which simplifies synchronization but couples rollout and training resources, limits heterogeneous deployment, and constrains independent scaling. To this end, we introduce DigenRL, a disaggregated RL framework for diffusion-based generative LLMs that supports flexible resource allocation, accommodates heterogeneous GPUs, and facilitates efficient task scheduling. To maximally reduce the execution bubbles in the disaggregated architecture, we propose: 1) a generation-axis pipeline (GAP) and time-step parallelism (TSP) in the diffusion architecture to enable finer-grained pipelining between rollout and training; 2) an elastic trainer-assisted generation (TAG) approach to enable the trainer GPU resources to dynamically assist in executing rollout generations; and 3) a tightly one-step constrained asynchronous strategy to further utilize the tail bubble in the pipeline. Extensive experiments are conducted on three hardware testbeds with 16-32 GPUs using HunyuanVideo-13B, Wan2.1-14B, FLUX.1-12B, and QwenImage-20B generative models. Experimental results show that DigenRL achieves 1.56-2.10x throughput improvements over state-of-the-art diffusion RL systems, veRL-Omni and GenRL.
