Search papers, labs, and topics across Lattice.
This paper introduces SLORR, a novel framework for in-training low-rank regularization that maintains model architecture while enabling effective weight compression. Unlike existing methods that require complex operations or additional parameters, SLORR employs stateless, GPU-friendly approximations to regularize weight matrices directly, achieving significant compressibility with minimal training overhead. Evaluations on various models, including ResNet and ViT architectures, demonstrate that SLORR can enhance model performance during compression with less than 8% additional training cost, particularly excelling in large language model pretraining scenarios.
SLORR achieves substantial model compressibility with under 1% training overhead, outperforming traditional regularization methods in preserving performance.
Low-rank factorization is widely used to compress neural networks, but modern models are often not naturally amenable to aggressive factorization without significant accuracy loss. Existing training-time low-rank regularizers can improve compressibility, but they often require SVDs of large weight matrices, modify the model architecture (introducing additional trainable parameters), or rely on stateful cached quantities. To address these limitations, we introduce SLORR, a simple, stateless, and architecture-preserving framework for in-training low-rank regularization, instantiated with two main variants based on the Hoyer sparsity metric and the nuclear norm. SLORR directly regularizes the original weight matrices using GPU-friendly approximations for the forward and backward passes of the regularizers, for which we provide approximation guarantees. We first evaluate SLORR on ImageNet-1K across short-horizon continued training of ResNet-50, ViT-B/16, and ViT-L/16, and pretraining of ResNet-18, where SLORR induces compressibility while introducing less than 8% training overhead. We further evaluate SLORR-Hoyer in LLM pretraining at 135M and 560M scales: SLORR-trained compressed models preserve performance substantially better than unregularized models while adding less than 1% average training overhead.
