Search papers, labs, and topics across Lattice.
This paper addresses the challenge of scheduling pipeline parallelism for LLM training across geo-distributed clusters with heterogeneous bandwidth and electricity costs. They propose BACE-Pipe, a framework that dynamically prioritizes jobs based on computation time and network utilization, and employs a bandwidth-aware pathfinder to identify feasible cross-region pipeline paths. By optimizing GPU placement based on electricity prices and mitigating head-of-line blocking, BACE-Pipe achieves significant reductions in both job completion time and electricity costs.
Training LLMs across regions doesn't have to break the bank: BACE-Pipe cuts job completion time by up to 65% and electricity costs by 30% by intelligently scheduling pipeline stages across different geographic locations.
The rapid evolution of large language models (LLMs) has made geographically distributed training necessary due to GPU scarcity within a single cloud region. In such cross-region settings, Pipeline Parallelism (PP) is communication-efficient, yet scheduling PP remains challenging under heterogeneous inter-region bandwidth and regional electricity prices. Existing schedulers are either delay-first, incurring high electricity cost, or cost-first, relying on rigid resource allocation that prolongs Job Completion Time (JCT). They are also ineffective at optimizing execution order in multi-tenant environments, where long-running and bandwidth-intensive jobs can cause head-of-line (HoL) blocking and degrade overall performance. To this end, we propose BACE-Pipe, a bandwidth-aware and cost-efficient pipeline scheduling framework for LLM training across geo-distributed clusters. BACE-Pipe first introduces a dynamic job prioritization mechanism that optimizes execution order by jointly considering job characteristics (e.g., computation time) and real-time network utilization. It then employs a bandwidth-aware pathfinder to identify feasible cross-region pipeline paths that satisfy communication constraints, thereby preventing communication from stalling the pipeline. Among all feasible paths, a cost-minimizing allocator determines the optimal GPU placement strategy by preferentially assigning resources to regions with lower electricity prices. Consequently, BACE-Pipe mitigates HoL blocking, improves resource utilization, and simultaneously reduces both JCT and total electricity cost. Extensive simulations show that BACE-Pipe reduces average JCT by 27.9%--64.7% and total electricity cost by 12.6%--30.6% compared with state-of-the-art baselines.
