Search papers, labs, and topics across Lattice.
The paper introduces SORT (Systematically Optimized Ranking Transformer), a Transformer-based ranking model tailored for industrial-scale recommender systems, addressing challenges of feature sparsity and low label density. SORT incorporates request-centric sample organization, local attention, query pruning, and generative pre-training, along with refinements to tokenization, multi-head attention, and feed-forward networks. Online A/B testing in e-commerce scenarios demonstrates that SORT significantly improves key business metrics (orders, buyers, GMV) while also reducing latency and increasing throughput.
A Transformer-based ranking model can boost e-commerce orders by 6.35% while halving latency, thanks to optimizations targeting feature sparsity and low label density.
While Transformers have achieved remarkable success in LLMs through superior scalability, their application in industrial-scale ranking models remains nascent, hindered by the challenges of high feature sparsity and low label density. In this paper, we propose SORT (Systematically Optimized Ranking Transformer), a scalable model designed to bridge the gap between Transformers and industrial-scale ranking models. We address the high feature sparsity and low label density challenges through a series of optimizations, including request-centric sample organization, local attention, query pruning and generative pre-training. Furthermore, we introduce a suite of refinements to the tokenization, multi-head attention (MHA), and feed-forward network (FFN) modules, which collectively stabilize the training process and enlarge the model capacity. To maximize hardware efficiency, we optimize our training system to elevate the model FLOPs utilization (MFU) to 22%. Extensive experiments demonstrate that SORT outperforms strong baselines and exhibits excellent scalability across data size, model size and sequence length, while remaining flexible at integrating diverse features. Finally, online A/B testing in large-scale e-commerce scenarios confirms that SORT achieves significant gains in key business metrics, including orders (+6.35%), buyers (+5.97%) and GMV (+5.47%), while simultaneously halving latency (-44.67%) and doubling throughput (+121.33%).
