Search papers, labs, and topics across Lattice.
This paper introduces NSDS, a data-free layer-wise mixed-precision quantization framework that improves upon existing methods by considering both numerical and structural sensitivity when allocating bit precision. NSDS decomposes each layer into distinct operational roles and quantifies their sensitivity from both numerical and structural perspectives, aggregating these into a unified layer-wise metric. Experiments show NSDS outperforms existing methods across various models and tasks without calibration data, enabling effective compression under extreme low-bit settings.
Achieve better compression in low-bit quantization by considering not just numerical sensitivity, but also the structural role of each layer.
Layer-wise mixed-precision quantization (LMPQ) enables effective compression under extreme low-bit settings by allocating higher precision to sensitive layers. However, existing methods typically treat all intra-layer weight modules uniformly and rely on a single numerical property when estimating sensitivity, overlooking their distinct operational roles and structural characteristics. To address this, we propose NSDS, a novel calibration-free LMPQ framework driven by Numerical and Structural Dual-Sensitivity. Specifically, it first mechanistically decomposes each layer into distinct operational roles and quantifies their sensitivity from both numerical and structural perspectives. These dual-aspect scores are then aggregated into a unified layer-wise metric through a robust aggregation scheme based on MAD-Sigmoid and Soft-OR to guide bit allocation. Extensive experiments demonstrate that NSDS consistently achieves superior performance compared to various baselines across diverse models and downstream tasks, without relying on any calibration data.
Tsinghua AI