Search papers, labs, and topics across Lattice.
This paper addresses the inefficiencies in standard A/B-testing protocols by identifying how overlapping actions between treatment and control policies contribute to unnecessary noise in outcome measurements. By framing the treatment assignment as a meta-policy and utilizing $\Delta$-Off-Policy Estimation methods, the authors provide a way to obtain unbiased estimates of average treatment effects while reducing variance. Their approach not only recovers standard practices but also demonstrates improved performance in scenarios where policies share common support, leading to more efficient online experimentation.
Overlapping actions in A/B tests inflate noise and confidence intervals, but a novel meta-policy approach can significantly reduce this variance and enhance treatment effect estimation.
Online controlled experiments are the gold standard for hypothesis testing in online platforms. Notwithstanding their ubiquity, they are notoriously expensive to run, and issues of variance hamper statistical power in assessing treatment effects. While standard variance reduction techniques leverage model-based control variates to reduce outcome noise, they remain agnostic to potential structural relationships between competing policies. In this work, we identify a critical inefficiency in the standard A/B-testing protocol: when a treatment and control policy agree on an action, the resulting outcome contributes noise but no signal regarding the treatment effect -- unnecessarily inflating confidence intervals. We propose a novel experimental protocol that exploits this policy overlap to accelerate experimentation. The key insight is to frame the randomised treatment assignment mechanism as a meta-policy, and leverage $\Delta$-Off-Policy Estimation methods to obtain unbiased estimates for average treatment effects. We prove analytically that our approach recovers standard A/B-testing practices in the general case, but that its variance scales with the divergence between policies rather than raw outcome variance. Hence, we dominate the standard Difference-in-Means estimator whenever policies have common support, and the improvement is strict whenever the overlap region contributes non-zero residual variance. Empirical results corroborate these theoretical insights -- holding promise for significant impact on the real-world evaluation of recommender systems, information retrieval pipelines, and large language model interfaces.
