Search papers, labs, and topics across Lattice.
This paper addresses the challenge of generating high-quality training data for dense retrieval models in e-commerce sponsored search without relying on click signals or manual annotation. By leveraging the disagreement among heterogeneous retrieval systems, the authors develop a structured training signal approach that incorporates multi-channel retrieval mining, graded-relevance annotation, and a progressive curriculum for organizing over 240 million training examples. The resulting two-tower BERT model outperforms the click-trained baseline by 5.1% NDCG@10 and demonstrates significant improvements in ad performance metrics during online testing.
Leveraging structured training signals from retrieval system disagreements, this approach boosts retrieval performance and ad engagement metrics significantly without traditional click-based methods.
How can we generate high-quality training data for dense retrieval models at production scale, without relying on click signals or manual annotation? This question is critical for e-commerce sponsored search, where click-based training suffers from position bias and tail-query sparsity, and manual labeling at the scale of hundreds of millions of query-item pairs is economically infeasible. Our work is driven by the following insight: heterogeneous retrieval systems disagree on most items they retrieve, and this disagreement creates a natural source of structured training signal -- easy positives where all systems agree, hard positives that only lexical systems find, and hard negatives that fool exactly one system. As our key novelty, we combine three ideas into an end-to-end pipeline: (a) multi-channel retrieval mining with rank metadata from three production systems, (b) graded-relevance annotation by a calibrated three-model cascade ) that reaches 89.1% agreement with trained human annotators, and (c) three-stage progressive curriculum training that organizes 240M+ training examples across five difficulty levels. We deploy the trained two-tower BERT model on Walmart's sponsored search and evaluate it against 30K queries labeled by trained third-party human annotators. First, we show that the system achieves +5.1% NDCG@10 over the click-trained production baseline, with the largest gain on tail queries . Second, we show that embarrassing retrievals (rating 0) drop from 8.7% to 3.5%. Third, a two-week online A/B test with tens of millions of ad requests per arm confirms +2.80% ad spend, +1.4% CTR, +2.8% eCPM, and +2.9% click conversion rate. Overall, our work provides a practical and scalable blueprint for replacing click-based training with structured LLM-annotated supervision in production retrieval systems.
