Search papers, labs, and topics across Lattice.
This paper investigates the scaling laws for compressing large language models (LLMs) specifically for domain-specific applications, focusing on quantitative finance. It reveals that while in-domain task quality degrades predictably with increased compression, general knowledge performance collapses much earlier, with the format of supervision being a critical factor in this tradeoff. A novel blended chain-of-thought supervision loss is introduced, which effectively recovers general knowledge during the distillation process, leading to practical insights and recommendations for LLM deployment in resource-constrained environments.
Compression strategies can preserve general knowledge in LLMs better than previously thought, thanks to innovative supervision techniques.
Large Language Models (LLMs) achieve strong performance across a growing range of domains, yet their scale poses deployment challenges in applications where latency and cost constraints are critical. This paper derives empirical scaling laws for domain-specific LLM compression, quantifying how in-domain and general knowledge performance scale with dataset size, compression ratio, supervision format, and iterative pruning schedule. Using quantitative finance as our application domain, we compare logit-based and LoRA-based distillation under iterative structural pruning, introducing a blended chain-of-thought supervision loss that stabilizes KL-divergence distillation over reasoning traces. In-domain task quality degrades predictably under compression while general-knowledge benchmarks collapse well before the same point; supervision format is the key driver of this tradeoff, with chain-of-thought supervision actively recovering general knowledge that pruning erases. We release the headline dataset FinHeadlineMix, scaling law results, and practical recommendations to provide a reusable framework for domain-specific compression decisions.
