Search papers, labs, and topics across Lattice.
This paper introduces a fine-grained post-training quantization strategy for large vision language models (LVLMs) called Quantization-aware Integrated Gradients (QIG). QIG leverages integrated gradients to quantitatively evaluate token sensitivity at the token level, capturing both inter-modality and intra-modality dynamics, unlike existing methods that operate at the modality level. Experiments on multiple LVLMs under W4A8 and W3A16 settings demonstrate that QIG improves accuracy across models and benchmarks with negligible latency overhead, achieving, for example, a 1.60% accuracy improvement on LLaVA-onevision-7B under 3-bit weight-only quantization.
Quantizing large vision-language models just got a whole lot better: a new token-level sensitivity metric closes the accuracy gap with full-precision models by up to 1.6% in 3-bit weight-only quantization.
Large Vision Language Models (LVLMs) have achieved remarkable success in a range of downstream tasks that require multimodal interaction, but their capabilities come with substantial computational and memory overhead, which hinders practical deployment. Among numerous acceleration techniques, post-training quantization is a popular and effective strategy for reducing memory cost and accelerating inference. However, existing LVLM quantization methods typically measure token sensitivity at the modality level, which fails to capture the complex cross-token interactions and falls short in quantitatively measuring the quantization error at the token level. As tokens interact within the model, the distinction between modalities gradually diminishes, suggesting the need for fine-grained calibration. Inspired by axiomatic attribution in mechanistic interpretability, we introduce a fine-grained quantization strategy on Quantization-aware Integrated Gradients (QIG), which leverages integrated gradients to quantitatively evaluate token sensitivity and push the granularity from modality level to token level, reflecting both inter-modality and intra-modality dynamics. Extensive experiments on multiple LVLMs under both W4A8 and W3A16 settings show that our method improves accuracy across models and benchmarks with negligible latency overhead. For example, under 3-bit weight-only quantization, our method improves the average accuracy of LLaVA-onevision-7B by 1.60%, reducing the gap to its full-precision counterpart to only 1.33%. The code is available at https://github.com/ucas-xiang/QIG.
