Search papers, labs, and topics across Lattice.
This paper introduces TSVD, a method for efficient LLM pre-training that maintains low-rank and strict orthonormality of weight matrices throughout training. TSVD uses a spectral energy-based heuristic for adaptive rank selection and caching mechanisms to enforce orthonormality. Experiments show that TSVD matches or exceeds the performance of full-parameter baselines while significantly reducing compute, offering a practical and scalable path to efficient LLM pre-training.
LLMs can now be pre-trained with significantly less compute thanks to a new method that adaptively maintains low-rank and orthonormal weight matrices.
The massive scaling of Large Language Models (LLMs) has made pretraining increasingly cost-prohibitive. While low-rank representation and orthonormal weight matrices could in principle reduce parameter counts and computational overhead, most existing methods rely on static rank selection and do not enforce weight orthonormality due to high computational cost. This paper introduces TSVD, a framework that maintains low rank and strict orthonormality throughout the training process. It utilizes a spectral energy-based heuristic for adaptive rank selection, and a caching mechanisms to maintain orthonormality. Theoretical analysis justifies the advantage of the approach in pretraining dynamics and experiments across various model scales demonstrate that it is effective empirically. TSVD matches or exceeds the performance of full-parameter baselines while significantly reducing compute requirements. The approach thus offers a well-founded, practical, and scalable path toward efficient high-performance LLM pretraining.
