Search papers, labs, and topics across Lattice.
This paper introduces FastTPS, an optimized method designed to enhance the token phase of inference in decoder-only large language models (LLMs) on AI accelerators. By implementing a reloading-free KV Cache concatenation, a highly-efficient 'RoPE' attention mechanism, and a fused MLP with fine-grain pipeline scheduling, FastTPS achieves a remarkable 6x speed improvement while maintaining 93% peak memory bandwidth utilization. This advancement addresses critical memory bottlenecks associated with long-sequence inputs, making LLM inference more efficient on modern AI hardware.
FastTPS accelerates LLM inference by 6x while preserving memory efficiency, revolutionizing how we handle long-sequence inputs on AI accelerators.
The popularity of large language models (LLMs) escalates an ongoing demand for effective inference. However, due to the sequential processing of tokens during the token phase in decoder-only LLMs inference, the inherent low parallelism leads to reduced throughput and suboptimal utilization of the computing units on artificial intelligence (AI) accelerators, particularly when handling long-sequence inputs that impose significant memory overhead. Recently, many reported methods have been developed as potential solutions, since they emerge with numeric deviation. This paper presents FastTPS, a high performance and low-precision loss method for accelerating the token-phase in LLM inference on general AI accelerators which includes three key components: (1) AI accelerator-enabled reloading-free KV Cache concatenation which decreases memory access overhead as well as enables full fusion of Attention, (2) high-efficiency and high-accuracy 'RoPE' attention based on the tiling optimized FLAT, and (3) highly-fused MLP with fine-grain pipeline scheduling. Our results confirm that FastTPS significantly alleviates memory bottlenecks in the token phase, delivering a 6x speed improvement (compared to none-fusion) on an AMD Ryzen AI 300 series NPU with BF16 precision while sustaining 93% peak memory bandwidth utilization during Phi3-mini-4k-instruct inference.
