Search papers, labs, and topics across Lattice.
This paper introduces LowRank-SSM, a hardware-software co-design framework that optimizes the deployment of State Space Models (SSMs) on FPGAs by explicitly treating projection rank as a design variable. By employing post-training truncated SVD for weight decomposition and a greedy rank-allocation algorithm, the approach minimizes memory usage while adhering to accuracy constraints. The implementation on Xilinx Versal VC1902 achieves a throughput improvement of 2.19x and energy efficiency improvement of 2.03x over state-of-the-art methods, demonstrating significant advancements in resource-constrained environments.
Achieving 2.19x throughput and 2.03x energy efficiency improvements on FPGA for SSMs by optimizing projection rank could redefine performance benchmarks in latency-sensitive applications.
State Space Models(SSMs) such as Mamba and Mamba-2 achieve linear-time autoregressive inference, making them attractive for latency-sensitive and resource-constrained deployment. Yet their large input and output projection layers impose quadratic weight memory and off-chip bandwidth costs that bottleneck practical FPGA deployment, accounting for over 60% per-token runtime at sequence lengths of 1,024 and beyond. Existing accelerators reduce this overhead through quantization or activation sparsity, but none treat projection rank as an explicit hardware design variable, leaving a systematic accuracy-throughput trade-off unexplored. We present LowRank-SSM, a hardware-software co-design framework that closes this gap. On the software side, we decompose the input and output projection weights via post-training truncated SVD and introduce a greedy bandwise rank-allocation algorithm that searches for the per-band rank vector that minimizes weight storage while respecting a user-specified accuracy constraint. On the hardware side, we map the resulting factored projections onto a fully-pipelined accelerator on an FPGA, featuring a dual-path projection(low-rank path and full-rank path), a fused selective-scan unit, and five independent AXI master bundles that saturate DDR4 bandwidth without bus contention. A per-band runtime rank mask enables mixed-rank execution across all 64 layers with zero architectural overhead. On Xilinx Versal VC1902 at 400 MHz, the deployed mixed-rank INT8 design achieves 7.89~tokens/s, representing a ${2.19\times}$ throughput improvement and ${2.03\times}$ energy-efficiency improvement over SOTA at comparable power and accuracy.
