Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces DASH-Q, a post-training quantization (PTQ) method for large language models that uses a diagonal Hessian approximation and iterative weighted least squares to improve robustness at ultra-low bit widths. DASH-Q mitigates noisy curvature estimates, a common problem in low-bit PTQ, by discarding cross-channel dependencies and focusing on preserving salient feature power. Experiments across five LLMs demonstrate that DASH-Q outperforms existing PTQ methods in the ultra-low bit regime, achieving an average zero-shot accuracy improvement of 7.01% and up to 14.01% over strong baselines.
Quantizing LLMs to ultra-low bit widths doesn't have to tank performance: DASH-Q achieves surprisingly high accuracy by ditching noisy cross-channel dependencies in Hessian-based quantization.
Large Language Models (LLMs) are widely used across many domains, but their scale makes deployment challenging. Post-Training Quantization (PTQ) reduces memory footprint without retraining by leveraging a small calibration set. Recent Hessian-based PTQ methods compensate quantization error via cross-channel dependencies, but such approaches degrade at low bit-widths due to noisy curvature estimates from limited calibration data. We propose DASH-Q, a robust PTQ framework using diagonal Hessian approximation and iterative weighted least squares. By discarding noise-prone dependencies, DASH-Q filters sampling noise while prioritizing the preservation of salient feature power. We outperform other PTQ baselines in ultra low-bit regime, improving zero-shot accuracy by 7.01% on average and up to 14.01% over the strongest baselines across five baseline LLM models, while showing robust and stable performance with very small calibration data.