Track trials, findings, and clinical evidence as it lands.
Search papers, labs, and topics across Lattice.
This paper analyzes the self-distillation mechanism in CTR prediction models, revealing that it mathematically reduces to a label loss on logit-averaged predictions plus a disagreement penalty between dropout copies. FLMNet leverages this insight by training without sampling, incorporating a closed-form approximation of the penalty when feature gating is absent. The resulting single-tower model achieves state-of-the-art accuracy on five benchmarks, demonstrating the efficacy of both the penalty and ensemble distillation.
Self-distillation in CTR prediction simplifies to a logit-averaged loss plus a disagreement penalty, enabling sample-free training and boosting single-tower accuracy.
Click-through rate (CTR) prediction underpins recommender systems and online advertising, where high-dimensional categorical features interact in combinatorially many ways. Several recent CTR models, such as FinalNet and QNN, report higher accuracy when a network also learns from its own averaged predictions, for example by running the same network under different dropout patterns and training each run toward the average. Why this self-distillation helps, and whether its random sampling is needed, remains unclear. This work shows that as the number of dropout patterns grows without bound, the objective reduces exactly to the label loss of the logit-averaged prediction plus a penalty on how much the dropout copies of the network disagree. When dropout does not affect how a network gates its features, this penalty can be approximated in closed form, and FLMNet is designed on this principle and trains without sampling. Networks trained from different seeds disagree in the same form, and distilling several of them into one network recovers part of their ensemble gain. On five benchmarks against seventeen baselines, the resulting models take the top two places in accuracy with a single network at inference, and the gains of the penalty and of distillation, where measurable, largely add up.