Search papers, labs, and topics across Lattice.
This paper introduces Libra, an innovative resource management system designed for agentic reinforcement learning (RL) in large language models (LLMs). By addressing the challenges of long-tailed trajectory distributions and asymmetrical compute patterns, Libra employs a global resource planner and a causality-driven multi-level feedback queue to optimize GPU allocation and scheduling. The results demonstrate that Libra can achieve up to 3.0x higher throughput and 2.5x faster convergence in reward compared to existing baselines, significantly enhancing the efficiency of RL post-training processes.
Achieving up to 3.0x higher throughput in agentic RL, Libra redefines resource management for large language models under complex workloads.
Reinforcement learning (RL) has become a standard post-training paradigm for large language models (LLMs), extending beyond preference alignment to complex reasoning and multi-turn agentic behaviors. In agentic RL, the rollout stage generates trajectories while invoking tools, producing long-tailed and non-stationary workloads that challenge conventional resource-management assumptions. Three fundamental challenges arise. First, due to the long-tail distribution, a small fraction of trajectories dominates rollout makespan. Second, rollout and training exhibit strong asymmetry in compute patterns, memory demands, and sensitivity to sequence length. Third, as the RL policy evolves, the trajectory-length distribution drifts over time, rendering any static resource split progressively suboptimal. We present Libra, which introduces two core mechanisms. The first is a periodic global resource planner that jointly optimizes GPU allocation across rollout and training clusters. It leverages an elastic hybrid pool to enable lightweight, non-blocking worker reallocation between stages. The second is a causality-driven multi-level feedback queue (C-MLFQ) scheduler, which routes requests to heterogeneous rollout buckets based on causal signals derived from tool-return outcomes, rather than relying on fragile length predictions. Evaluated on 48 A800 GPUs, Libra achieves up to 3.0$\times$ higher throughput and converges up to 2.5$\times$ faster in reward compared to the baselines.
