Track new results across pure and applied mathematics.
Search papers, labs, and topics across Lattice.
ETH introduces Heterogeneous Pipeline Parallelism (HPP) to address the memory and coarse-grained balancing limitations of existing systems by enabling fine-grained data balance across heterogeneous pipelines. The ETH system achieves this through a hierarchical workload balance framework and block-wise gradient aggregation to mitigate communication overhead, alongside optimizations for reduced configuration search. Experimental results show ETH delivers up to 2x training speedup and significantly lowers configuration generation overhead compared to prior methods.
Fine-grained data balancing in heterogeneous pipelines unlocks 2x training speedups for large models by overcoming memory bottlenecks and coarse workload assignments.
Current heterogeneity-aware training systems for large models balance workloads by assigning unequal numbers of layers across pipeline stages, a strategy aimed at equalizing execution time through a unique pipeline configuration. However, this approach fails to address critical memory constraints and suffers from inherently coarse balancing granularity. In this paper, we propose Heterogeneous Pipeline Parallelism (HPP), a novel approach enabling efficient and fine-grained data balance across heterogeneous pipelines. To address HPP's key challenges, prohibitive search overhead for optimal configurations and high communication overhead from asymmetric All-Reduce, we design the ETH training system. ETH employs a hierarchical workload balance framework that jointly optimizes balance across heterogeneous pipelines and within individual pipeline stages. To mitigate asymmetric communication costs, ETH implements block-wise gradient aggregation. Furthermore, it integrates specialized optimizations that dramatically reduce HPP configuration search overhead. Evaluations demonstrate ETH achieves up to 2x training speedup while reducing configuration generation overhead by up to 56x compared to state-of-the-art systems. The code is available at https://github.com/YuMJie/ETH.