Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
Nereus is a cost-aware runtime for LLM reinforcement learning post-training that dynamically reconfigures distributed tensor, pipeline, and data parallelism across multi-model, multi-stage pipelines. Addressing the issue where shifting sequence lengths and memory pressure render static distributed execution plans bottlenecked or prone to out-of-memory errors, it models the trade-off between plan efficiency and transition costs online. Evaluated on runs scaling up to 1,024 GPUs, Nereus reduces average step latency by 27.7% with sub-0.1% transition overhead, achieving 1.10–1.47× throughput improvements over Verl and up to 7.27× over OpenRLHF.
Static parallelism layouts leave massive post-training efficiency on the table, but dynamically re-slicing distributed model states across GPUs cuts RL step latency by 28% with less than 0.1% transition overhead.
Reinforcement learning (RL) post-training for large language models (LLMs) coordinates multiple models across generation, inference, and training on GPU clusters. Several factors may change during a run, including resource availability, sequence length, memory pressure, and stage bottlenecks. As a consequence, an execution plan that was initially suitable can then become slow or even infeasible over time. However, adapting a job whose models share GPUs entails significant challenges: deciding whether a new plan is worth the transition cost, reusing the job's distributed state, and coordinating GPU transfers across models and stages. Nereus targets these challenges as a cost-aware runtime that adapts RL post-training jobs into efficient execution plans. Its low-overhead controller selects a memory-feasible global plan and admits the transition using a cost model calibrated against the running job. To estimate and execute a transition, Nereus represents the distributed state of each replica of a model-stage (one model in one stage) as an Elastic Model Unit. It then employs a global transition graph to order the transformations and GPU transfers of these units. In a trace built from real data, online TP/PP adaptation reduces average step latency by 27.7% relative to the initial fixed TP/PP layout with DP scaling. In a 1,000-step run reaching 1,024 GPUs, six transitions consume 0.079% of total run time. Nereus improves end-to-end 8B PPO throughput by 2.14--7.27times over OpenRLHF and by 1.10--1.47times over Verl across diverse clusters.