Search papers, labs, and topics across Lattice.
This paper introduces heterogeneous parallelism, a novel training abstraction for multimodal LLMs that decouples the parallelism strategies of different modules (e.g., encoder and LLM). By allowing independent layouts and rank placements for each module, the method addresses the limitations of LLM-centric parallelism, particularly when dealing with long contexts and diverging encoder/LLM scales. Experiments demonstrate that heterogeneous parallelism can improve TFLOPS/GPU by up to 49.3% in colocated settings and aggregate token throughput by up to 13.0% in non-colocated settings, while maintaining loss convergence.
Stop forcing your multimodal encoders to inherit suboptimal LLM parallelism strategies: heterogeneous parallelism unlocks up to 49% higher TFLOPS/GPU.
Foundation model training is becoming multimodal, from post-training pipelines to large-scale pretraining. As modality coverage broadens, context windows grow, and encoder LLM scales diverge, a single LLM-centric TP/CP/PP/DP/EP layout increasingly limits throughput. This coupling forces encoders to inherit LLM-driven sharding and placement choices that can add communication, limit encoder parallelism, or constrain the LLM schedule; the mismatch is most pronounced at long contexts, where LLM context parallelism is needed for the fused multimodal sequence but encoder inputs remain bounded. We present heterogeneous parallelism for multimodal large language model training, an abstraction that lets modules in one end-to-end graph use independent layouts and rank placements, supporting colocated execution on shared GPUs and non-colocated execution on disjoint rank sets. The key challenge is preserving boundary tensor semantics across independent layouts: forward activations must be materialized for the destination layout, while backward gradients must be routed back to the source layout. We address this with boundary communicators that implement forward and backward layout transforms, plus scheduling extensions for both placement modes. We evaluate optimized homogeneous, colocated heterogeneous, and non-colocated heterogeneous configurations across multimodal workloads and GPU scales to characterize when added layout and placement freedom exposes a better operating point. Across this sweep, colocated heterogeneity improves TFLOPS/GPU by up to 49.3%, while non-colocated heterogeneity improves aggregate token throughput by up to 13.0% and TFLOPS/GPU by up to 9.6%. We validate loss convergence parity against homogeneous baselines and release the system as an open-source Megatron-LM extension.
