Search papers, labs, and topics across Lattice.
This paper introduces SkipOPU, an FPGA-based overlay processor designed to accelerate LLM inference with dynamic computation allocation by exploiting token and layer contribution variance. SkipOPU decouples reduction operations in nonlinear modules for fusion with linear operations, uses a float-fixed hybrid execution PE array with DSP overpacking, and implements a proactive on-chip KV history buffer to reduce off-chip memory access. Experiments on an AMD U280 FPGA show SkipOPU achieves 1.23x-3.83x better bandwidth efficiency than GPUs and other FPGAs, and reduces KV storage overhead by up to 25.4%.
SkipOPU achieves up to 3.83x speedup in LLM inference by dynamically allocating computation on an FPGA, outperforming GPUs and other FPGAs.
Large language models (LLMs) have achieved remarkable performance across a wide range of tasks, but their inference efficiency remains a critical bottleneck due to rapidly growing parameters. Recent advances in dynamic computation allocation address this challenge by exploiting the highly uneven contributions of different tokens and layers, enabling selective execution that significantly reduces redundant computation while preserving model accuracy. However, existing hardware platforms and accelerators are primarily optimized for uniform, static execution, limiting their ability to efficiently support such dynamic inference patterns. In this work, we propose SkipOPU, an FPGA-based overlay processor that dynamically allocates computation across tokens and layers with high flexibility through a lightweight routing mechanism. First, we decouple reduction operations from element-wise computation in nonlinear modules and perform reductions incrementally, which enables both stages to be fused with adjacent linear operations (router or matrix multiplication) for effective latency hiding. Second, motivated by asymmetric sensitivity to numerical precision between activation and weight, we design a PE array that efficiently supports float-fixed hybrid execution. A novel DSP overpacking technique is introduced to maximize hardware utilization while minimizing resource overhead. Finally, we develop a proactive on-chip KV history buffer that exploits cross-layer KV invariance of pruned tokens, eliminating irregular HBM accesses during decoding and supplementing off-chip bandwidth through high-locality on-chip reuse. Experimental results demonstrate that SkipOPU on an AMD U280 FPGA outperforms GPU and other FPGA-based accelerators by 1.23x-3.83x in bandwidth efficiency for LLMs inference with dynamic computation allocation and can reduce up to 25.4% KV storage overhead across varying sequence lengths.
