Search papers, labs, and topics across Lattice.
This paper establishes a formal equivalence between online A/B testing's Difference-in-Means estimator and off-policy evaluation's Inverse Propensity Scoring with optimal control variates, bridging the gap between these traditionally separate domains. It further demonstrates that regression adjustment methods like CUPED, CUPAC, and ML-RATE are structurally equivalent to Doubly Robust estimation. This unification provides a more comprehensive understanding of variance reduction techniques and can inform both online and off-policy experimentation.
Online A/B testing's classic Difference-in-Means estimator is just off-policy Inverse Propensity Scoring in disguise.
Continuous and efficient experimentation is key to the practical success of user-facing applications on the web, both through online A/B-tests and off-policy evaluation. Despite their shared objective -- estimating the incremental value of a treatment -- these domains often operate in isolation, utilising distinct terminologies and statistical toolkits. This paper bridges that divide by establishing a formal equivalence between their canonical variance reduction methods. We prove that the standard online Difference-in-Means estimator is mathematically identical to an off-policy Inverse Propensity Scoring estimator equipped with an optimal (variance-minimising) additive control variate. Extending this unification, we demonstrate that widespread regression adjustment methods (such as CUPED, CUPAC, and ML-RATE) are structurally equivalent to Doubly Robust estimation. This unified view extends our understanding of commonly used approaches, and can guide practitioners and researchers working on either class of problems.
