Search papers, labs, and topics across Lattice.
The paper investigates the bottleneck of sign bits in sub-bit model compression, revealing that learned sign matrices are spectrally similar to random Rademacher matrices and resist low-rank approximation. It identifies a "sign lock-in" phenomenon where initial weight signs persist throughout training due to rare near-zero crossings. The authors formalize this with a stopping-time analysis of sign flips under SGD noise and propose a gap-based initialization coupled with an outward-drift regularizer to reduce the effective flip rate, improving sub-bit compression.
Weight signs in neural networks are far more stable and predictable than previously thought, bottlenecking extreme compression efforts.
Sub-bit model compression seeks storage below one bit per weight; as magnitudes are aggressively compressed, the sign bit becomes a fixed-cost bottleneck. Across Transformers, CNNs, and MLPs, learned sign matrices resist low-rank approximation and are spectrally indistinguishable from an i.i.d. Rademacher baseline. Despite this apparent randomness, most weights retain their initialization signs; flips primarily occur via rare near-zero boundary crossings, suggesting that sign-pattern randomness is largely inherited from initialization. We formalize this behavior with sign lock-in theory, a stopping-time analysis of sign flips under SGD noise. Under bounded updates and a rare re-entry condition into a small neighborhood around zero, the number of effective sign flips exhibits a geometric tail. Building on this mechanism, we introduce a gap-based initialization and a lightweight outward-drift regularizer, reducing the effective flip rate to approximately $10^{-3}$ with only about a one-point increase in perplexity.
