Search papers, labs, and topics across Lattice.
This paper challenges the conventional wisdom that A/B testing is superior to offline evaluation methods for algorithm selection, revealing that A/B testing can actually lead to higher selection error rates due to its reliance on sample mean estimators that lack positive correlation. The authors introduce a novel estimator that incorporates a hypothetical middle algorithm to induce positive correlation, thereby enhancing the accuracy of algorithm selection in A/B testing. Experiments show that this new approach achieves comparable selection accuracy to traditional methods while requiring only half the data, significantly improving efficiency.
A/B testing may be more error-prone than offline evaluations, but a new estimator leveraging a hypothetical middle algorithm can turn this on its head.
A/B testing is the gold standard for selecting the better algorithm in online services. While offline evaluation has attracted attention as a safer alternative due to the high experimental costs and the potential risk of degrading user experience and revenue in A/B testing, it is widely recognized that the estimation accuracy of offline evaluation is substantially lower. As a result, final selection decisions are typically made through A/B testing. Contrary to this conventional view, we reveal a counterintuitive phenomenon in which A/B testing can produce a higher algorithm selection error rate than offline evaluation. This occurs because the sample mean estimator used in A/B testing does not induce positive correlation, which is crucial for reducing critical selection errors, namely underestimating the truly superior algorithm and overestimating the truly inferior one. In contrast, offline evaluation methods unintentionally generate this beneficial correlation by relying on shared offline data when estimating and comparing the performance of multiple algorithms. Building on this insight, we propose an estimator that intentionally induces positive correlation to improve algorithm selection in A/B testing. The key idea is to introduce a hypothetical middle algorithm and to estimate the performance difference between algorithms A, M, and B in a stepwise manner using shared data at each step. This approach enables the application of offline evaluation techniques in each step, thereby inducing positive correlation and reducing critical selection errors. Furthermore, we derive the optimal middle algorithm regarding the resulting variance and analyze its advantages over existing methods through bias-variance analysis. Experiments on real-world data demonstrate that our estimator achieves the same selection error rate as existing approaches while using only one half of the A/B testing data.
