Search papers, labs, and topics across Lattice.
This paper introduces Uni-SFU, an algorithm-hardware co-design framework that optimizes the implementation of nonlinear activation functions in deep neural networks by using mixed-degree polynomial approximations across nonuniform segments. By jointly searching for the best approximation strategy and hardware configuration, Uni-SFU significantly reduces hardware redundancy and improves precision, achieving a Mean Squared Error (MSE) of less than 8.22x10^-8 with only a 1.02% degradation in top-1 accuracy compared to traditional floating-point implementations. The design demonstrates a compact silicon area of 6,800 um虏 in GF 22nm CMOS technology, outperforming state-of-the-art methods in both accuracy and area efficiency.
Achieving a staggering reduction in hardware footprint while maintaining near-perfect accuracy, Uni-SFU redefines the efficiency of activation function implementations in neural networks.
Nonlinear activation functions are essential to modern deep neural networks (DNNs), but their hardware evaluation places significant pressure on the special-function units (SFUs) of GPUs and custom accelerators. Therefore, piecewise polynomial approximations are commonly used within allowed error bounds to improve computational efficiency. However, existing techniques often approximate each activation function in isolation using fixed-degree polynomials and uniform segments, leading to hardware redundancy and sub-optimal precision. To address these limitations, we present Uni-SFU, an algorithm-hardware co-design framework that jointly optimizes approximation accuracy and silicon area for a diverse set of activation functions. Uni-SFU leverages a joint search across all target functions to assign mixed-degree polynomials to nonuniform segments, guided by an RTL-derived area cost model. This approach identifies a unified hardware configuration to implement the target activation functions under given accuracy constraints. Validated across over 700 neural network variants and three Natural Language Processing (NLP) models, Uni-SFU achieves a superior Mean Squared Error (MSE) below 8.22x10^-8, limiting top-1 accuracy degradation to within 1.02% compared to floating-point baselines. The proposed design occupies only 6,800 um2 in GF 22nm CMOS technology, achieving a superior trade-off between silicon area and system-level accuracy compared to SOTA counterparts.
