Search papers, labs, and topics across Lattice.
This paper introduces a Hierarchical Sequence-aware Parallelism (HSAP) framework that addresses the limitations of existing sequence parallelism methods in handling hybrid-context packed sequences, which often lead to cross-contamination in attention computation. By employing a Sequence-Aware Parallelism algorithm and Just-In-Time (JIT) compilation, the authors optimize tensor transmission and attention computation across device groups, significantly enhancing the efficiency of pretraining and fine-tuning large language models. Experimental results show that HSAP outperforms state-of-the-art sequence parallelism approaches across various performance metrics, highlighting its effectiveness in improving model training processes.
HSAP achieves unprecedented efficiency in training large language models by eliminating attention cross-contamination while maintaining high parallelism.
In this paper, we aim to combine the advantages of existing sequence parallelism paradigms and overcomes their drawbacks, the most serious of which is the incapability to correctly compute causal attention on the hybrid-context packed sequences, in a stronger sequence parallelism framework. The practical technique of packing sequences for efficiently pretraining and fine-tuning large language models causes cross-contamination problem in attention computation, which can be effectively solved when no parallelism in the sequence length dimension is taken. However, in sequence parallelism, existing approaches either ignore the scenario of hybrid-context sequences or conversely sacrifice and limit parallelism degree for supporting the scenario. To this end, we innovatively propose an efficient Sequence-Aware Parallelism algorithm to conquer the obstacles of intensive tensor transmission and partial attention computation across multiple device groups. Our algorithm utilizes JIT (Just-In-Time) compilation to optimize the communication strategy of all device groups in NCCL level. Further, we integrate existing sequence parallelism paradigms into a Hierachical Sequence-Aware Parallelism framework which benefits from our sequence-aware algorithm. We additionally elaborate on the memory and communication overhead management of the hierachical framework to optimize its performance. Through multiple experiments, we demonstrate that our proposed approach outperform other state-of-the-arts sequence parallelism approches in multiple metrics.
