Search papers, labs, and topics across Lattice.
The paper introduces MultiFaceted Learnable Index (MFLI), a novel retrieval paradigm for large-scale recommendation systems that jointly learns item embeddings and hierarchical indices via residual quantization to eliminate the need for ANN search at serving time. MFLI constructs a multifaceted hierarchical codebook and co-trains it with item embeddings, enabling efficient multifaceted indexing and real-time updates. Experiments on real-world datasets demonstrate significant improvements in recall, cold-content delivery, and semantic relevance compared to existing ANN-based methods, along with improved online engagement and serving efficiency.
Ditch ANN search altogether: MFLI learns a hierarchical index alongside item embeddings, boosting recall by up to 11.8% and cold-content delivery by 57.29% in large-scale recommender systems.
Approximate nearest neighbor (ANN) search is widely used in the retrieval stage of large-scale recommendation systems. In this stage, candidate items are indexed using their learned embedding vectors, and ANN search is executed for each user (or item) query to retrieve a set of relevant items. However, ANN-based retrieval has two key limitations. First, item embeddings and their indices are typically learned in separate stages: indexing is often performed offline after embeddings are trained, which can yield suboptimal retrieval quality-especially for newly created items. Second, although ANN offers sublinear query time, it must still be run for every request, incurring substantial computation cost at industry scale. In this paper, we propose MultiFaceted Learnable Index (MFLI), a scalable, real-time retrieval paradigm that learns multifaceted item embeddings and indices within a unified framework and eliminates ANN search at serving time. Specifically, we construct a multifaceted hierarchical codebook via residual quantization of item embeddings and co-train the codebook with the embeddings. We further introduce an efficient multifaceted indexing structure and mechanisms that support real-time updates. At serving time, the learned hierarchical indices are used directly to identify relevant items, avoiding ANN search altogether. Extensive experiments on real-world data with billions of users show that MFLI improves recall on engagement tasks by up to 11.8\%, cold-content delivery by up to 57.29\%, and semantic relevance by 13.5\% compared with prior state-of-the-art methods. We also deploy MFLI in the system and report online experimental results demonstrating improved engagement, less popularity bias, and higher serving efficiency.
Meta AI