Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces SISA, a systolic array architecture that partitions a traditional square array into independently scheduled horizontal rectangular slabs to address the underutilization of processing elements (PEs) when executing GEMM operations on input-dependent and skewed matrices common in LLMs. SISA enables parallelism for small or skewed matrix shapes while maintaining full-array operation for large GEMMs. The proposed architecture achieves up to 8.52x speedup and 93% energy-delay-product (EDP) reduction compared to a monolithic SA with the same number of PEs on representative LLMs.
LLMs' skewed matrix shapes are leaving your systolic arrays underutilized, but SISA's partitioned architecture offers up to 8.52x speedup and 93% EDP reduction.
The currently dominant AI/ML workloads, such as Large Language Models (LLMs), rely on the efficient execution of General Matrix-Matrix Multiplication (GEMM) operations. Thus, most systems are equipped with dedicated matrix hardware accelerators based on square Systolic Arrays (SAs) of Processing Elements (PEs). While this organization was effective for traditional Deep Neural Networks (DNNs), LLMs introduce input-dependent and highly skewed matrices, leading to underutilized SA resources. To address this challenge, we propose SISA (Scale-In Systolic Array), a novel SA architecture that partitions the traditional square array into horizontal rectangular slabs. With minimal overhead, SISA exposes parallelism through independently scheduled slabs for efficient execution of small or skewed matrix shapes, while retaining full-array operation for large GEMMs. SISA achieves up to 8.52x speedup and 93% energy-delay-product (EDP) reduction for representative LLMs compared to a state-of-the-art monolithic SA with the same number of PEs.