Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
Tangram is a novel system designed to optimize LLM parallelization in heterogeneous GPU clusters by decoupling the parallelization planning process from GPU heterogeneity. By leveraging the concept of homogeneous GPU islands and model partitioning, Tangram allows existing parallelizers to efficiently manage the complexities introduced by varied GPU types and interconnects. The results show that Tangram can achieve up to 2.3x higher training throughput compared to existing heterogeneous parallelizers like Metis and Sailor, significantly enhancing the scalability of LLM training jobs.
Achieving up to 2.3x higher training throughput, Tangram transforms how we approach parallelization in heterogeneous GPU clusters.
The scale of LLM training jobs requires parallelization planning over large GPU clusters. Due to different GPU types and interconnects added over time, these GPU clusters are increasingly heterogeneous. Automatic LLM parallelizers can search for parallelization plans but face an exploding search space with heterogeneous GPUs. To make search tractable in heterogeneous GPU clusters, parallelizers often omit types of parallelism (e.g., expert parallelism) or memory-saving techniques (e.g., ZeRO), which results in worse plans. We describe Tangram, a system that enables the use of existing heterogeneity-unaware LLM parallelizers in heterogeneous GPU clusters by decoupling parallelization planning from GPU heterogeneity. For this, Tangram exploits two insights: (1) since bulk purchases result in sets of GPUs with similar compute, memory, and connectivity, Tangram can expose such homogeneous GPU islands to existing parallelizers; and (2) parallelizers commonly first partition models and then parallelize partitions. Tangram can compose such model slices, assigned to GPU islands, into work-balanced pipelines for high throughput. Tangram integrates with existing parallelizers through a narrow API, which relies on the enumeration of model-slice/island pairs. Tangram achieves up to 2.3x higher training throughput than current heterogeneous parallelizers (Metis and Sailor) and scales to large GPU clusters by pruning enumerated plans.