Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces Taylor-Calibrate, a novel initialization method designed to enhance the conversion of pretrained Transformer models into hybrid linear attention models, specifically Gated DeltaNet (GDN) students. By leveraging Taylor-guided teacher attention statistics, the method effectively sets crucial model parameters, ensuring that the converted models start in a more favorable dynamical regime. The results show that Taylor-Calibrate significantly improves zero-shot performance, achieving up to an 88x increase in efficiency and requiring 4.9x to 9.2x fewer training tokens compared to traditional conversion methods.
Achieving up to 88x efficiency gains, Taylor-Calibrate transforms the way we initialize hybrid linear attention models, drastically reducing the training burden.
Hybrid linear attention models offer an appealing path to faster long-context inference: they reduce the quadratic cost and KV-cache burden of full softmax attention while retaining much of the quality of Transformer models. A practical way to obtain such models is to convert a pretrained Transformer instead of pretraining a new architecture from scratch, but this conversion is still brittle. Simply copying the teacher attention projections into a Gated DeltaNet (GDN) student does not specify the new recurrent decay, write, and output-gating dynamics. As a result, the converted model often starts in a poor dynamical regime and must spend many distillation tokens repairing initialization rather than learning the remaining teacher behavior. We propose Taylor-Calibrate, a lightweight initialization method for hybrid GDN students. The method uses Taylor-guided teacher attention statistics to set the value projection, memory timescale, write gates, and output gate, then applies a short per-layer alignment step to match each converted layer to the teacher output. Across four teacher settings and three retained-layer policies, Taylor-Calibrate gives substantially stronger zero-shot students, with up to an 88x improvement in a representative ablation, and reaches matched recovery targets with 4.9x--9.2x fewer training tokens than naive conversion.
BAIR