Search papers, labs, and topics across Lattice.
This paper introduces FlexViT, a reconfigurable FPGA-based accelerator designed to optimize Vision Transformer (ViT) inference on edge devices, addressing the challenges posed by the heterogeneous architecture of modern ViT models. By employing a hardware-software co-design that integrates a high-throughput INT8 GEMM engine with a runtime im2col transformation, FlexViT efficiently manages the diverse tensor shapes of both fully connected and convolutional layers. The implementation on a PYNQ-Z2 FPGA demonstrates significant performance improvements, achieving up to 2.74x speedup on accelerator-executed layers and up to 1.40x end-to-end speedup compared to traditional CPU execution.
FlexViT achieves up to 2.74x speedup for Vision Transformer inference on edge devices, revolutionizing the deployment of complex models in resource-constrained environments.
Deploying Vision Transformer (ViT) models on edge platforms remains challenging due to their high computational demands and the architectural heterogeneity of modern hybrid ViT models, which incorporate both fully connected and convolutional layers. This heterogeneity leads to significant variation in tensor shapes, requiring flexible and efficient FPGA-based acceleration. In this paper, we present FlexViT, a reconfigurable FPGA accelerator for efficient ViT inference on resource-constrained edge devices. Built on the SECDA-TFLite framework, FlexViT employs a hardware-software co-design approach that maps both fully connected and convolutional layers onto a unified high-throughput INT8 GEMM engine using a runtime im2col transformation. To efficiently support diverse layer configurations, we propose a dual-mode dataflow that dynamically switches between input and weight reuse by reconfiguring the compute array at runtime. We further introduce a depth-first tiling strategy that completes accumulation in a single pass, eliminating off-chip partial-sum transfers and reducing memory bandwidth requirements. We implement FlexViT on a PYNQ-Z2 FPGA and evaluate it across a representative set of ViT models. FlexViT achieves up to 2.74x speedup on accelerator-executed layers, translating into up to 1.40x end-to-end speedup compared to CPU-only execution. The code is available at: https://github.com/gicLAB/FlexViT
