Search papers, labs, and topics across Lattice.
This paper introduces C-PTQ, a novel post-training quantization method that addresses the performance degradation in multimodal large language models (MLLMs) caused by sensitive outlier channels. By employing a Fisher-weighted objective to harmonize task-specific loss perturbation with quantization error, C-PTQ optimizes channel-wise scaling more effectively than existing methods. Experimental results on multiple benchmarks show that C-PTQ achieves state-of-the-art performance without the need for auxiliary modules, ensuring both efficiency and accuracy in model deployment.
C-PTQ reveals that harmonizing task sensitivity with quantization error can dramatically enhance MLLM performance without additional complexity.
Multimodal large language models (MLLMs) require huge memory and computational costs, which limits their practical deployment. Post-training quantization (PTQ) techniques offer an efficient solution for model compression and inference acceleration. Yet, the quantized model faces performance degradation due to outlier channels, which are highly sensitive to quantization and substantially impair activation fidelity and task accuracy. To protect these salient channels during quantization, existing PTQ methods leverage modality- or token-level metrics to guide channel-wise scaling (CWS) of LLM decoders. However, these orthogonal measurements fail to capture channel-wise impacts on task-specific loss, and the misalignment between importance and scaling factors ultimately leads to suboptimal performance. To address this issue, we propose C-PTQ, a unified channel-wise PTQ method that harmonizes task-specific loss perturbation and quantization error. Motivated by second-order derivatives, we design a Fisher-weighted objective as a tractable Hessian approximation, seamlessly injecting task sensitivity into the scaling process. Notably, we achieve state-of-the-art performance without auxiliary modules like LoRA, thereby maintaining high efficiency. Experiments on Qwen2.5VL, InternVL2 and LLaVA-OV across 8 benchmarks demonstrate our effectiveness in both weight-only and weight-activation settings.
