Search papers, labs, and topics across Lattice.
This paper introduces Request-Oriented Compute Sharing (ROCS), a novel paradigm for enhancing the efficiency of large-scale recommendation systems by optimizing how user requests interact with candidate items. By deferring interactions and leveraging shared request-side features, ROCS achieves significant improvements in inference efficiency while maintaining or even enhancing prediction quality. Experimental results demonstrate that ROCS can yield up to a 3x increase in queries per second (QPS) for retrieval models and a 0.5% relative LogLoss improvement in short-form video ranking, showcasing its practical benefits in real-world applications.
ROCS can triple the throughput of recommendation systems without sacrificing prediction accuracy, revolutionizing how we handle user requests in large-scale settings.
Modern recommendation models gain prediction quality by scaling feature-interaction and sequence modules, but production cost constraints cap how far systems can scale. In this work, we propose Request-Oriented Compute Sharing (ROCS), a modeling and inference paradigm that exploits a unique property of recommendation inference: each user request is evaluated against many candidates, while request-side features are shared across candidates. ROCS defers request-candidate interactions as late as possible, isolates candidate-dependent representations, and evaluates substantial portions of the model once per request rather than once per candidate, significantly improving inference efficiency while maintaining or improving prediction quality. To realize this paradigm, we develop Generalized Layer Masking (GLM) to enforce candidate isolation in feature-interaction architectures, and Deep Cross Attention (DCA) to extend request-oriented sharing to sequence architectures. To support efficient GPU deployment, we co-design In-Kernel Broadcast Optimization (IKBO) that significantly accelerates ROCS model execution. Experiments on public benchmarks show that ROCS consistently improves the quality-efficiency tradeoff across recommendation backbones. On production-scale workloads, ROCS achieves up to a 3x QPS improvement on retrieval models without quality degradation and a 0.5% relative LogLoss improvement with a 50% QPS gain on a short-form video ranking model. ROCS has been deployed across large-scale recommendation systems spanning ads and organic surfaces, retrieval and ranking stages, and more than two orders of magnitude in inference complexity, delivering significant online gains at reduced infrastructure cost.
Meta AI