Search papers, labs, and topics across Lattice.
This paper formalizes external validity as the probability that independent replications of a study yield results with a maximum mean discrepancy (MMD) below a tolerance, using kernel methods to define a space of results and an appropriate kernel. An asymptotic power-law relation between study size and MMD quantiles is derived, enabling estimation of externally valid study sizes from preliminary experiments. The framework is demonstrated on rankings with ties and other research questions, illustrating its broad applicability.
External validity is quantified by the probability of low MMD between independent study replications, enabling principled sample size determination for generalizable experimental findings.
The external validity of a study is the extent to which its findings generalize to unseen experimental conditions, such as datasets. We formalize the results of a study together with its research question, and define external validity as the probability that two independent and same-sized replications of the study produce results whose maximum mean discrepancy (MMD) is below a given tolerance. This approach, centered on the choice of a space of results and of an appropriate reproducing kernel for the MMD, is flexible enough to be deployed on multiple different types of results and research questions. Additionally, we establish an asymptotic power-law relation between the size of a study and the quantiles of the MMD, which we use to estimate the desired size of an externally valid study from a small-scale preliminary study. We deploy our framework to rankings with ties and discuss kernels suited to different research questions. Experiments on existing case studies and on a simulated newly designed study illustrate the applicability of the framework. We release our code as part of the genexpy Python module.
