Search papers, labs, and topics across Lattice.
This paper introduces the memory layer, an innovative in-model key-value embedding cache that is co-trained with recommendation models to eliminate discrepancies between training and serving item representations. By ensuring a single source of truth for item embeddings, the memory layer achieves 100% prediction coverage and significantly enhances embedding freshness, reducing the training-serving Normalized Entropy gap by up to 86%. Deployed on Instagram Reels, this approach not only doubles recall for fresh content but also reduces computational costs by 30% without impacting serving performance.
Co-training item embeddings in recommendation systems can boost prediction accuracy and engagement while slashing computational costs by 30%.
Early ranking stages in recommendation systems precompute item embeddings and cache them in-model for scoring within strict latency constraints. Because this cache exists only at serving time, outside the training loop, training and serving use different item representations, a structural discrepancy that limits quality and adds operational fragility. We show that co-designing the training and serving paths removes this representation discrepancy at its source. We introduce the memory layer, an in-model key-value embedding cache co-trained with the model: the item tower writes embeddings during training and the model reads them at serving, one source of truth for item representations by construction. Always-on embeddings cover items not yet cached, so every item receives a prediction, and the design consolidates three separate trainer-to-predictor update paths into a single self-contained pipeline. Deployed in production on Instagram Reels, the memory layer raises prediction coverage from 96% to 100%, improves embedding freshness from $O(5\text{ min})$ to $O(20\text{ s})$, and narrows the training-serving Normalized Entropy (NE) gap by up to 86%, yielding over $2\times$ recall for the freshest content and a 5-6% cold start engagement lift. Because embeddings are produced during training, the system needs no separate bulk-evaluation or publish-time recomputation, cutting training-and-publish computational cost by 30% at neutral serving computational cost.
