Search papers, labs, and topics across Lattice.
This paper introduces Privileged Self-Distillation (PSD), a novel framework for sequential recommendation that leverages future interactions as a source of privileged information during training while maintaining alignment with inference-time predictions. By employing two attention masks鈥攐ne for future-aware supervision and another for prefix-only deployment鈥擯SD allows for richer training signals without altering the architecture or inference costs. Experiments demonstrate that this method consistently enhances recommendation performance across various benchmarks and model architectures.
Future interactions can be harnessed to significantly boost recommendation accuracy without compromising inference efficiency.
Sequential recommenders are commonly trained with one-hot next-item labels under a causal (prefix-only) objective aligned with inference. While deployment-compatible, this supervision offers little insight into relative preferences among non-target items. Yet logged interaction sequences contain an additional supervisory source: interactions following the target often reveal how user intent evolves, making the target easier to interpret. We treat these future interactions as training-only privileged information, available during learning but not at inference. This raises a natural question: can future interactions provide richer supervision while keeping training aligned with inference-time prediction? We propose Privileged Self-Distillation (PSD), a framework that separates learning-time information from inference-time information. PSD applies two attention masks to the same backbone: a future-aware view yields a privileged teacher distribution conditioned on past and future interactions, while a prefix-only view yields the student distribution used for deployment. Distilling the privileged distribution converts future interactions into training-only supervision rather than inference-time inputs. Since both views share a backbone, the teacher's advantage is purely informational, not architectural, removing the need for a separately pretrained teacher and letting its supervision adapt as the student evolves. PSD further uses an advantage-reachability gate to focus distillation on teacher signals likely supported by the observed prefix, along with a momentum-averaged teacher for stable targets. The framework is optimized end-to-end in a single stage, leaving the deployed model and inference cost unchanged. Experiments across public benchmarks and diverse backbones show consistent improvements.
