Search papers, labs, and topics across Lattice.
This paper introduces a novel post-training quantization (PTQ) framework specifically designed for visual autoregressive models (VAR), addressing significant challenges such as high reconstruction errors and sampling frequency discrepancies. By implementing a shift-and-sum quantization method and a resampling strategy for calibration data, the authors effectively enhance the accuracy of attention-value products and align codebook entry frequencies with predicted probabilities. Experimental results demonstrate that their approach achieves consistent improvements across various VAR architectures, setting a new benchmark in the field of PTQ for visual models.
Shift-and-sum quantization reduces reconstruction errors in visual autoregressive models, achieving state-of-the-art performance in post-training quantization.
Post-training quantization (PTQ) enables efficient deployment of deep networks using a small set of data. Its application to visual autoregressive models (VAR), however, remains relatively unexplored. We identify two key challenges for applying PTQ to VAR: (i) large reconstruction errors in attention-value products, especially at coarse scales where high attention scores occur more frequently; and (ii) a discrepancy between the sampling frequencies of codebook entries and their predicted probabilities due to limited calibration data. To address these challenges, we propose a PTQ framework tailored for VAR. First, we introduce a shift-and-sum quantization method that reduces reconstruction errors by aggregating quantized results from symmetrically shifted duplicates of value tokens. Second, we present a resampling strategy for calibration data that aligns sampling frequencies of codebook entries with their predicted probabilities. Experiments on class-conditional image generation, inpainting, outpainting, and class-conditional editing show consistent improvements across VAR architectures, establishing a new state of the art in PTQ for VAR.
