Search papers, labs, and topics across Lattice.
This study introduces a novel approach to quantization-aware training (QAT) for State Space Models (SSMs) by leveraging a pretrained checkpoint, significantly reducing the token budget required for training. By employing grouped QAT with knowledge distillation from a frozen FP16 teacher, the authors compress the Mamba-2 model while maintaining competitive zero-shot accuracy. The findings reveal that this method not only enhances efficiency but also uncovers a new instability phenomenon, zero-ratio collapse, which challenges traditional training assumptions for SSMs.
Ternary SSMs can achieve competitive performance without the costly overhead of from-scratch training, slashing token requirements by 1,000x.
State Space Models (SSMs) such as Mamba-2 offer linear-time inference but their memory footprint limits edge deployment. Prior ternary SSM work (Slender-Mamba) trains from scratch on 150B tokens; we show a pretrained checkpoint suffices, reducing the marginal token budget by 1,000x. Using grouped quantization-aware training (QAT) with knowledge distillation from a frozen FP16 teacher, we compress Mamba-2 1.3B to 3.61x (2,687 to 744 MB) and achieve 48.1% zero-shot accuracy (7-task average) in just 102M tokens (4 GPU-hours, single H100) -- approaching Bi-Mamba's 48.4% (within +/-0.9pp CI). This QAT-from-pretrained setting reveals zero-ratio collapse, a novel instability caused by learnable quantization scales that does not arise in from-scratch training. We further show that post-hoc correction strategies effective for Transformers fail for SSMs due to error accumulation through the recurrence. These results demonstrate that ternary SSMs do not require expensive from-scratch training: QAT from pretrained checkpoints with KD is a data-efficient alternative.
