Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
The paper introduces Dynamic Hybrid Parallelism (DHP), a novel parallelism strategy for training MLLMs on heterogeneous multimodal datasets that addresses load imbalance and suboptimal hardware utilization inherent in static parallelism approaches. DHP dynamically reconfigures communication groups and parallelism degrees during training, generalizing non-power-of-two parallelism and employing a polynomial-time algorithm for near-optimal strategy generation. Experiments show DHP achieves up to 1.36x speedup compared to Megatron-LM and DeepSpeed, while maintaining near-linear scaling efficiency on NPU clusters.
MLLM training gets a 1.36x speed boost with Dynamic Hybrid Parallelism (DHP), which adaptively optimizes parallelism strategies to handle the data heterogeneity that plagues multimodal datasets.
Scaling long-context capabilities is crucial for Multimodal Large Language Models (MLLMs). However, real-world multimodal datasets are extremely heterogeneous. Existing training frameworks predominantly rely on static parallelism strategies, which suffer from severe load imbalance, redundant communication, and suboptimal hardware utilization under data heterogeneity. In this work, we propose Dynamic Hybrid Parallelism (DHP), an efficient parallelism strategy that adaptively reconfigures communication groups and parallelism degrees during MLLM training. We generalize the non-power-of-two parallelism degrees and develop a polynomial-time algorithm to generate near-optimal parallelism strategies with only millisecond-level overhead per training batch. DHP is able to maintain high hardware efficiency even under extreme data variability. Experimental results demonstrate that DHP significantly outperforms Megatron-LM and DeepSpeed, achieving up to 1.36 $\times$ speedup in training throughput while maintaining near-linear scaling efficiency across large-scale NPU clusters.