Search papers, labs, and topics across Lattice.
This paper investigates the limitations of perplexity-based sensitivity rankings in mixed-precision quantization (MPQ) for large language models, revealing a phenomenon termed the Perplexity Illusion, where important layers for perplexity do not correlate with those critical for reasoning performance. It introduces the Alignment-Diversity Tradeoff, highlighting that using only target-task calibration data can hinder post-quantization performance, while incorporating general-domain data enhances robustness. The proposed Task-Aware Sensitivity Analysis (TASA) framework optimizes calibration-data composition and bit allocation, achieving significant performance improvements, including a 20-point increase on GSM8K with a 3.5-bit model compared to a 4-bit baseline.
Calibration-data composition can dramatically enhance quantization performance, with 3.5-bit models outperforming traditional 4-bit baselines by over 20 points.
Mixed-precision quantization (MPQ) has become a key technique for deploying large language models under stringent memory and compute constraints. We first identify a phenomenon that we term the Perplexity Illusion: layers ranked as important by perplexity-based sensitivity show little rank correlation with those that are most influential for complex reasoning performance, with Kendall $蟿\approx 0$ in our analysis. We further reveal an Alignment-Diversity Tradeoff: using only target-task calibration data can degrade post-quantization performance, whereas incorporating general-domain data stabilizes sensitivity estimation and improves robustness across tasks. Based on these observations, we propose TASA (Task-Aware Sensitivity Analysis), a two-level framework that jointly optimizes calibration-data composition and mixed-precision bit allocation. Specifically, TASA searches for a calibration-data mixture using a training-free gradient-trace alignment criterion, and then aggregates perplexity and reasoning-oriented sensitivity signals to guide both inter-layer and intra-layer bit allocation. Experiments on LLaMA-3-8B and Qwen2.5-7B reveal a precision inversion: appropriately allocated 3.5-bit models can match or surpass less task-aware 4-bit baselines. At an average precision of 3.5 bits, TASA matches or outperforms several competitive 4-bit uniform baselines in aggregate accuracy, and improves over the strongest W3 baseline on GSM8K by more than 20 absolute points on LLaMA-3-8B. These results show that calibration-data composition substantially affects task-sensitive quantization, a factor underexplored in prior work.
