Search papers, labs, and topics across Lattice.
This study investigates continual pre-training (CPT) as a method for adapting large language models (LLMs) to specialized domains, revealing that while singular value spectra remain largely unchanged, adaptation is primarily influenced by alterations in singular vectors. The authors introduce a head importance criterion, demonstrating that up to 60% of attention-head updates can be discarded without compromising performance, and selectively rewinding low-importance heads enhances benchmark accuracy by up to 4%. Additionally, they uncover a phenomenon of domain connectivity, allowing for smooth interpolation between CPT checkpoints, and release the Diffract toolkit for spectral analysis of large models.
Up to 60% of attention heads can be pruned during domain adaptation without sacrificing performance, revealing a new avenue for efficient model tuning.
We study continual pre-training (CPT) as a mechanism for adapting general-purpose large language models to specialized domains: mathematics, instruction, code, and natural text. Using singular value decomposition of weight matrices, we find that CPT leaves singular value spectra largely invariant, with adaptation driven mainly by changes in singular vectors. An analysis of attention-head projection matrices reveals strong, domain-dependent head heterogeneity, which we exploit to define a head importance criterion: up to 60% of head updates can be removed without measurable quality loss. Selectively rewinding low-importance heads to their pre-trained state improves benchmark accuracy by up to 4% versus the fully trained baseline. Finally, we identify domain connectivity - linear interpolation between CPT checkpoints yields smooth domain-quality interpolation without notable degradation on either domain - and release Diffract, an open-source toolkit for scalable spectral analysis of billion-parameter models.
