Search papers, labs, and topics across Lattice.
This paper introduces ADAPT, a novel framework that enables the efficient generation of a continuum of language model variants by amortizing the distillation process across both model size and post-trained variant. By employing a two-phase distillation procedure and weight-delta initialization, ADAPT allows for the creation of $L \times K$ models from a single distillation run, significantly reducing the computational burden associated with producing multiple model sizes. The key result is that this approach not only streamlines the model development process but also enhances adaptive model-size selection, optimizing the compute-accuracy trade-off for long-form reasoning tasks.
Amortizing distillation across model size and variant can produce a continuum of optimized LLMs with a single distillation run, revolutionizing model deployment efficiency.
Practical deployment of large language models (LLMs) requires families of post-trained variants---instruction-tuned, reasoning-tuned, and chat-style models---each at multiple sizes to meet diverse latency and memory budgets. Producing each (variant, size) pair independently is prohibitive, so model families typically span only a handful of coarse-grained sizes per post-trained variant. Boomerang distillation (Kangaslahti et al., 2026) reduces this cost along the size axis for base models. Through model size interpolation, it constructs models of intermediate sizes from a single teacher-student pair without additional training. However, it still treats each post-trained variant as a separate object of optimization. We introduce ADAPT---Amortized Distillation Across Post-Trained LLMs---a framework for amortizing distillation across both axes of a model family: size and post-training variant, producing $L \times K$ models for $L$ interpolated sizes across $K$ post-trained variants with a single distillation run. ADAPT combines two components. First, a two-phase distillation procedure constructs post-trained students through pre-training alignment and supervised fine-tuning distillation, enabling smooth size--performance interpolation on generation and reasoning tasks. Second, weight-delta initialization approximates this construction across post-trained variants by transferring the distillation-induced weight change from the base model to students initialized from different post-trained variants. The resulting continuum of interpolated models also enables adaptive model-size selection at inference time, improving the compute--accuracy trade-off for long-form reasoning tasks.
