Search papers, labs, and topics across Lattice.
This paper introduces the concept of impression share prediction as an offline evaluation task for ranking systems, addressing the limitations of standard offline metrics that fail to capture the downstream utility of impression distribution across objective buckets. By employing a structural causal model, the authors identify how model predictions and delivery capacity jointly influence impression allocation, allowing for a more accurate prediction of impression shares from early-interaction signals. The proposed statistical learning framework, utilizing Random Forests, demonstrates significant improvements in predictive accuracy, achieving a 49% reduction in L1 error over a constant baseline for training data and recovering an additional 22% L1 improvement in challenging evaluation scenarios.
Predicting impression distribution before online evaluation can drastically improve the effectiveness of ranking models, revealing hidden shifts that traditional metrics overlook.
Offline evaluation is a major gateway before online evaluation of ranking models in A/B testing. Standard offline metrics measure predictive accuracy, but are only a surrogate for downstream utility: a model can improve them while redistributing impressions across objective buckets in ways that degrade downstream utility. No offline method surfaces these impression share shifts before online evaluation. We propose \emph{impression share prediction} as an offline evaluation task: given a candidate ranking model, predict the distribution of impressions it would produce across objective buckets - impressions grouped by optimization goal (e.g., click, video view). The task is inherently counterfactual, since the candidate has never served live traffic. We propose a structural causal model of how model predictions and delivery capacity jointly determine impression allocation, and show the counterfactual effect is identified from observational data. Building on this, we develop a statistical learning framework that predicts impression shares from a candidate's early-interaction confidence signals and current system state, trained on historical data. On data from multiple ranking model families, a Random Forest reduces L1 error by 49\% over a constant baseline for models seen during training. For held-out models, evaluated by time since first appearance, the first hour is the closest analog to true online evaluation and the hardest: the Random Forest falls below the baseline because the capacity state still reflects the prior model. An encoder-conditioned architecture that simulates a 2-hour rollout over recent auction dynamics recovers $+$22\% L1 in this regime.
