Search papers, labs, and topics across Lattice.
This paper introduces a calibrated model cascade designed to enhance the efficiency of relevance annotation for sponsored search by leveraging progressively larger fine-tuned classifiers. The authors demonstrate that while fine-tuning significantly boosts accuracy by 20 points, the cascading approach effectively reduces compute costs by half without compromising accuracy. Additionally, the integration of per-class isotonic calibration yields a modest but statistically significant improvement, validating the system's effectiveness across six offline use cases with over 150 million processed annotations.
Cost-efficient relevance evaluation can be achieved without sacrificing accuracy, thanks to a novel calibrated model cascade that halves compute costs while enhancing annotation quality.
How can we generate high-quality relevance annotations at scale without the cost and delays of human labeling? Relevance annotations are the backbone of search ranking systems which is needed for training data preparation, NDCG evaluation, and root cause analysis. However, human annotation is slow and off-the-shelf LLMs suffer from accuracy on domain-specific tasks. We propose a calibrated model cascade, a systematic approach for cost-efficient offline relevance annotation by routing queries through progressively larger fine-tuned classifiers. Our central insight is that accuracy and cost are orthogonal optimizations: domain-specific fine-tuning drives accuracy, cascading drives cost, and per-class isotonic calibration adds a small but reliable gain on top. Our contribution is threefold: (a) we decompose the gains and show that fine-tuning contributes 20 accuracy points while cascading is approximately accuracy-neutral but halves compute cost, (b) we introduce per-class isotonic calibration as one component of the cascade, contributing a small but statistically significant gain (+0.6 points over the strongest calibration baseline), and (c) we validate the system in production across six offline use cases, processing 150M+ annotations and enabling faster experimentation cycles. Our work is a building block for scalable, high-quality offline annotation pipelines in search and advertising systems.
