Search papers, labs, and topics across Lattice.
This paper introduces I-LW-DETR, the first fully integer-only lightweight DETR that enables efficient deployment of Vision Transformer detectors on NPUs and microcontrollers. By innovating with a scale-preserving split convolution, a sign-dependent GELU approximation, and a constrained Shiftmax, the authors achieve an end-to-end integer implementation that maintains performance while significantly reducing model size and computational cost. Experimental results show that I-LW-DETR achieves a moderate accuracy degradation while decreasing model size by approximately 3.6 times and computational cost by over an order of magnitude across various scales.
Achieving fully integer-only inference for lightweight detection transformers could revolutionize their deployment in resource-constrained environments.
Vision Transformer detectors now approach the accuracy of CNNs but remain difficult to deploy on NPUs and microcontrollers because key components, including deformable attention, feature fusion, and nonlinear activation functions, are not natively compatible with integer arithmetic. Existing quantized detectors either retain operators such as Softmax, GELU, and LayerNorm or focus on heavyweight backbones, leaving lightweight detection transformers without an end-to-end integer implementation. We address this gap with I-LW-DETR, the first fully integer-only lightweight DETR, in which every operation in the forward pass, including transformer nonlinearities, is executed in integer arithmetic. I-LW-DETR is built upon three key components: a scale-preserving split convolution that assigns independent activation scale to each branch of the multi-scale projector; SD-ShiftGELU, a sign-dependent GELU approximation that preserves element-wise behavior while avoiding the accuracy degradation; and a constrained Shiftmax that maintains stable Softmax normalization. Experimental results demonstrate that the proposed quantization pipeline consistently produces efficient fully integer-only models across different model scales. Across all model scales, the proposed pipeline incurs only a moderate accuracy degradation while reducing the model size by approximately $3.6\times$ and the computational cost by more than one order of magnitude.
