Search papers, labs, and topics across Lattice.
This paper introduces a BRAM-free approximate attention-weighting unit designed for FPGA-based Vision Transformer (ViT) inference, addressing the significant hardware challenges posed by the softmax stage in self-attention mechanisms. By employing a 16-segment piecewise-linear function to approximate the natural exponential in softmax, the method maintains the pre-trained attention temperature without requiring model-specific recalibration. The implementation on a Xilinx Zynq-7020 shows that the design uses only 1444 LUTs and 77 DSPs while achieving a mere 0.20% absolute top-1 accuracy difference from the exact-softmax reference, highlighting its viability for energy-efficient deployment in edge-AI applications.
Achieving a 0.20% accuracy loss while eliminating BRAM usage could revolutionize the deployment of Vision Transformers on resource-constrained edge devices.
Vision Transformers have reshaped computer vision by using self-attention to capture global context across image regions. This makes them attractive for edge visual inspection and monitoring in applications such as renewable-energy infrastructure, industrial quality control, medical imaging, and autonomous-system sensing. However, deploying ViTs on small FPGAs remains challenging because the softmax stage in self-attention requires exponential evaluation and normalization, which are costly in hardware. Existing implementations often rely on CORDIC pipelines or BRAM-based look-up tables, increasing area and power consumption. This paper presents a BRAM-free approximate attention-weighting unit for FPGA-based ViT inference. The proposed design approximates the natural exponential in softmax using a 16-segment piecewise-linear function implemented entirely with distributed LUT fabric. Unlike base-2 approximations, the natural-exponential formulation preserves the pre-trained attention temperature and avoids model-specific recalibration. Implemented on a Xilinx Zynq-7020, the complete attention-row core uses 1444 LUTs, 77 DSPs, and no BRAM, while hardware-accurate emulation shows accuracy within a \(0.20\%\) absolute top-1 difference from the exact-softmax reference on ViT-family models. These results demonstrate the potential of the proposed core for energy-efficient ViT inference on resource-constrained edge-AI platforms.
