Track genomics, neuroscience and molecular biology.
Search papers, labs, and topics across Lattice.
This paper introduces MixFrag, a mixed-precision post-training quantization framework tailored for Vision Transformers (ViTs), which addresses the inefficiencies of uniform bit-width quantization by leveraging component-level fragility metrics. By measuring the Kullback-Leibler divergence between full-precision and quantized outputs, MixFrag allocates bit-widths adaptively through a Multiple-Choice Knapsack Problem formulation. Experimental results show that MixFrag not only maintains competitive classification performance on ImageNet-1K but also sets a new state-of-the-art in mixed-precision quantization for object detection and segmentation tasks on COCO, outperforming previous methods by up to 9.6 AP.
MixFrag redefines quantization efficiency by dynamically allocating precision based on the fragility of each transformer component, achieving unprecedented performance gains in resource-constrained environments.
Post-training quantization (PTQ) has emerged as an effective solution for deploying Vision Transformers (ViTs) on resource-constrained devices. However, existing PTQ methods typically employ uniform bit-widths across transformer components, overlooking their heterogeneous sensitivity to quantization and leading to inefficient precision allocation. In this paper, we propose {MixFrag, a fragility-guided mixed-precision PTQ framework for Vision Transformers. MixFrag first estimates component-level quantization fragility by measuring the Kullback--Leibler (KL) divergence between full-precision and isolated quantized output distributions using a small calibration set. It then formulates bit allocation as a Multiple-Choice Knapsack Problem (MCKP), enabling adaptive layer-wise precision assignment under a target bit budget. Extensive experiments on ImageNet-1K across multiple Vision Transformer architectures demonstrate that MixFrag achieves competitive classification performance under practical mixed-precision settings. Furthermore, evaluations on COCO object detection and instance segmentation show that MixFrag achieves state-of-the-art performance among existing mixed-precision PTQ methods, improving the previous best method by up to 9.6 AP under the challenging MP3/MP3 setting. Additional analyses validate the proposed fragility metric and demonstrate its strong correlation with the learned bit allocation. These results establish MixFrag as an effective framework for mixed-precision post-training quantization of Vision Transformers.