Search papers, labs, and topics across Lattice.
This paper introduces a novel preconditioning (PC) layer that employs polynomial weight parameterization to stabilize weight conditioning during the training of large language models (LLMs). By reshaping the singular-value spectrum of weight matrices, the PC layer enhances the training process, leading to improved performance in Llama-1B pre-training with both AdamW and Muon optimizers. Theoretical analysis confirms that bounding singular values across layers facilitates geometric convergence of gradient descent to global minima in specific deep linear networks, highlighting the method's robustness and efficiency.
Polynomial preconditioning can significantly enhance LLM training stability and performance without adding inference overhead.
We propose a preconditioning (PC) layer, a weight parameterization via polynomial preconditioner that ensures stable weight conditioning throughout LLM training. The PC module reshapes the singular-value spectrum of weight matrices via low-degree polynomial preconditioning. After training, the preconditioned weights can be merged back into the original architecture, incurring no inference overhead. We demonstrate the advantage of the proposed PC layer over standard transformers in Llama-1B pre-training, for both the AdamW and Muon optimizers. Theoretically, we justify this spectrum-control principle by proving that uniformly bounding each layer's singular values ensures geometric convergence of gradient descent to global minima, for certain deep linear networks. Our code is available at https://github.com/Empath-aln/PC-layer.
