Search papers, labs, and topics across Lattice.
This paper introduces Graft-Oriented Distillation (GOD), a novel framework that enhances generalization in sequential recommendation systems by employing a hybrid model approach. By selectively grafting trainable student components onto frozen teacher components, GOD provides component-level feedback that clarifies the sources of weak generalization, addressing issues related to sparse and noisy interaction histories. Experimental results demonstrate that GOD significantly outperforms existing state-of-the-art methods, achieving improvements of up to 13.92% across three real-world datasets.
Grafting trainable components onto frozen teacher models reveals the hidden pitfalls of generalization in sequential recommenders, leading to substantial performance gains.
Sequential recommenders often struggle with sparse and noisy histories, limiting generalization to unseen interactions. Knowledge distillation mitigates this by transferring dense supervision from a teacher to a student. However, most distillation methods run teacher and student independently, then match student outputs or representations to the teacher. Such supervision entangles student-component effects, blurring whether weak generalization stems from unreliable embeddings, overfitted encoding, or co-adaptation to sparse histories. In this paper, we propose Graft-Oriented Distillation (GOD), a component-level distillation framework for improved generalization through grafting. Grafting denotes replacing selected frozen-teacher components with trainable student counterparts to build hybrid source models. GOD uses these hybrid models to evaluate student embeddings with the teacher encoder and the student encoder with teacher embeddings, providing component-level feedback. At inference, GOD uses only the student, incurring no additional cost. Across three real-world datasets, GOD outperforms state-of-the-art baselines by up to 13.92%.
