Search papers, labs, and topics across Lattice.
This paper introduces a forward-pass-only sensitivity analysis framework using KL divergence to guide mixed-precision quantization for hybrid SSM-Transformer models. The method identifies quantization-sensitive components without backpropagation or retraining, addressing the challenge of deploying LLMs on resource-constrained edge devices. Experiments demonstrate that KL-based sensitivity rankings outperform MSE and SQNR, enabling near-FP16 perplexity with INT4-comparable model sizes and throughput on Intel Lunar Lake hardware.
Forget backpropagation: a forward-pass KL divergence metric pinpoints which parts of your SSM-Transformer hybrid model are most vulnerable to quantization, enabling efficient mixed-precision deployment on edge devices.
Deploying Large Language Models (LLMs) on edge devices faces severe computational and memory constraints, limiting real-time processing and on-device intelligence. Hybrid architectures combining Structured State Space Models (SSMs) with transformer-based LLMs offer a balance of efficiency and performance. Aggressive quantization can drastically cut model size and speed up inference, but its uneven effects on different components require careful management. In this work, we propose a lightweight, backpropagation-free, surrogate-based sensitivity analysis framework to identify hybrid SSM-Transformer components most susceptible to quantization-induced degradation. Relying solely on forward-pass metrics, our method avoids expensive gradient computations and retraining, making it suitable for situations where access to in-domain data is limited due to proprietary restrictions or privacy constraints. We also provide a formal analysis showing that the Kullback-Leibler (KL) divergence metric better captures quantization sensitivity for Language modeling tasks than widely adopted alternatives such as mean squared error (MSE) and signal-to-quantization-noise ratio (SQNR). Through extensive experiments on SSM and hybrid architectures, our ablation studies confirm that KL-based rankings align with observed performance drops and outperform alternative metrics. This framework enables the practical deployment of advanced hybrid models on resource-constrained edge devices with minimal accuracy loss. We further validate our approach with real-world on-device profiling on Intel Lunar Lake hardware, demonstrating that KL-guided mixed-precision achieves near-FP16 perplexity with model sizes and throughput competitive with Uniform INT4 on both CPU and GPU execution modes. Code is available at https://github.com/jasonkongie/kl-ssm-quant.
