Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper addresses the limitations of tensor parallelism (TP) in large language model (LLM) inference by introducing Albireo, a parallel inference system that optimizes the TP degree to maximize performance while minimizing non-scalable overheads. By overlapping scheduling and I/O with compute and sequence-parallel sampling, Albireo achieves significant improvements in throughput, latency, GPU utilization, and energy efficiency without altering model architectures. The results demonstrate that Albireo can achieve up to 1.9x higher throughput and 54% lower energy consumption compared to existing systems, making it a valuable advancement for LLM deployment.
Achieving up to 2x higher throughput in production by effectively managing tensor parallelism and non-scalable overheads could revolutionize LLM inference efficiency.
Deployers of online LLM services usually seek to maximize cluster-wide performance given a fixed number of GPUs. Tensor parallelism (TP) is necessary to fit modern models but scales sub-linearly as the TP degree t grows, due to cross-GPU communication and non-scalable runtime work, as predicted by Amdahl's Law. Conversely, increasing t improves memory efficiency and alleviates KV-cache contention and swapping. We identify and validate an empirical optimal TP degree t_e that balances these effects. We present Albireo, a parallel inference system that raises the attainable t_e by shrinking the non-scalable portion via overlap of scheduling and I/O with compute and sequence-parallel sampling, without changing model architectures. Across models and benchmarks, Albireo achieves up to 1.9x higher throughput, 48% lower latency, 28% higher GPU utilization, and 54% lower energy than vLLM; in production it yields up to 2x higher throughput.
Tsinghua AI