Search papers, labs, and topics across Lattice.
This paper introduces PACI, a novel asynchronous pipeline parallel training method that effectively manages weight inconsistency without the need for weight stashing or global synchronization. By employing local gradient accumulation to control version drift, PACI achieves bubble-free operation while maintaining high throughput and stability during training. The method demonstrates significant improvements in training time-to-accuracy, achieving up to 1.69 times faster results compared to traditional synchronous methods, while matching their performance in terms of stability and memory usage.
Weight inconsistency can be bounded for efficiency gains, leading to training time improvements of up to 1.69x without sacrificing stability.
Pipeline parallelism is essential for training large neural networks, but existing schedules trade off throughput, memory, and optimization consistency. Synchronous pipelines preserve forward/backward weight consistency but suffer from bubbles; asynchronous pipelines remove bubbles but introduce weight-version mismatch, typically requiring weight stashing, prediction, or correction mechanisms. We introduce PACI (Pipeline Asynchronous training with Controlled Inconsistency), a bubble-free asynchronous pipeline method that bounds forward/backward version drift without weight stashing, prediction, additional parameter copies, or global synchronization. The key idea is to use local gradient accumulation as a version-control mechanism: by slowing parameter-version evolution relative to pipeline delay, PACI limits the number of optimizer updates crossed by any micro-batch while preserving steady-state utilization. In GPT-style language-model pretraining, PACI matches the stability and final perplexity of synchronous 1F1B-flush, retains the same peak memory footprint, achieves fully utilized pipeline throughput, and improves training time-to-accuracy by up to 1.69times over the fastest flush baseline. These results show that forward/backward inconsistency need not be eliminated: when explicitly bounded, it can be safely traded for substantial efficiency gains.
