Search papers, labs, and topics across Lattice.
This paper investigates the impact of Depth Registers with a register-magnitude hinge loss (DR+sink) on W4A4 quantization in a 300M-parameter SwiGLU decoder-only language model, revealing that naive quantization methods significantly degrade performance. By employing DR+sink, the authors achieve a substantial reduction in validation perplexity from 1727 to 119 while maintaining comparable zero-shot capacity to FP16 models. The study further decomposes the sources of quantization error, identifying that the residual-axis readers are tightly controlled, while the bilinear input of the generators remains less constrained, highlighting the nuanced interplay between model components during quantization.
Depth Registers can slash W4A4 quantization error by nearly 14x, revealing critical insights into how model components contribute to performance degradation.
We study post-training W4A4 quantization in a controlled 300M-parameter SwiGLU decoder-only language model trained on 5B tokens of FineWeb-Edu, and ask which input-activation sites dominate the error. Naive round-to-nearest W4A4 collapses validation perplexity from FP16 23.6 to 1727. A simple residual-axis training-time intervention -- Depth Registers with a register-magnitude hinge loss (DR+sink) -- reduces this to 119 (about 14x) at matched FP16 PPL and matched zero-shot capacity, and composes with SmoothQuant to 39.9 PPL. The residual ~2 PPL gap to FP16 is the diagnostic core. We decompose W4A4 damage by input-activation site: the five trainable linears in a SwiGLU block split into residual-axis readers (qkv, w1, w3) and block-internal generators (o_proj, w2). Elementary norm arguments show residual-axis magnitude control bounds readers tightly but leaves w2's bilinear input bounded only by the trivial product of factor bounds; empirically, DR+sink collapses reader kurtosis while leaving generators essentially unchanged, and the reader-rescued W4A4 residue is flat at ~0.28 nats across three matched checkpoints with Delta-remove(w2) dominating. We present DR+sink as a training-time probe rather than a deployment proposal: a post-hoc alternative (Per-Linear QuaRot) nearly matches it on the reader axis. Full QuaRot -- adding online per-head value Hadamard plus online w2-input rotation -- does not close the gap either, directly testing the prediction that orthogonal rotation cannot bound the bilinear SwiGLU tail. Claims are specific to our 300M, 5B-token, single-seed setting, and our experiments do not isolate the partition from the hinge.
