Track synthesis, materials and catalysis as it is published.
Search papers, labs, and topics across Lattice.
This paper introduces a unified exponential framework, gamma-CUBV, that generalizes Cross Upper-Bound Validation (CUBV) to provide conservative upper bounds on true risk, encompassing various generalization gap settings including Hoeffding, Bernstein, PAC-Bayesian, and heterogeneous source fusion. It models dependence between cross-validation folds using a joint sub-Gaussian proxy matrix, yielding an effective number of folds that quantifies the diminishing statistical evidence from increased folds under strong dependence. The framework is extended to posterior distributions and weighted multi-source fusion, offering a principled validation criterion for heterogeneous small-sample applications by explicitly separating performance, uncertainty, dependence, complexity, and confidence.
K-fold cross-validation's statistical evidence diminishes with increasing folds when data heterogeneity creates strong inter-fold dependence, a phenomenon quantified by an effective number of folds.
K-fold cross-validation (CV) is widely used as evidence of out-of-sample performance, although folds are neither independent experiments nor equally informative under heterogeneous data. Cross Upper-Bound Validation (CUBV) replaces point-wise CV accuracy by conservative upper bounds on true risk. Here we generalise CUBV through a single exponential framework in which the moment-generating function of the generalisation gap is controlled by a cumulant envelope gamma(lambda). This yields a family of risk bounds covering Hoeffding-, Bernstein-, dependency-aware, PAC-Bayesian, and heterogeneous source-fusion settings. For K-fold CV, dependence between fold-wise gaps is modelled through a joint sub-Gaussian proxy matrix. Under equicorrelation, this gives an effective number of folds, Keff = K/[1+(K-1)rho], showing that increasing K does not necessarily increase statistical evidence when folds are strongly dependent. The framework is also extended to posterior distributions over predictors and weighted multi-source fusion, where weights are selected by minimising an upper bound on future risk rather than empirical error alone. Experiments with trained linear classifiers on heterogeneous multimodal Gaussian mixtures compare K-fold CV with full-sample resubstitution plus risk correction. Bounds are evaluated by coverage and tightness. In low-dimensional small-sample settings, K-fold partitioning can increase uncertainty because individual folds under-represent minority modes, while corrected resubstitution can remain valid and tighter; this effect disappears as sample size increases. Overall, gamma-CUBV separates observed performance, uncertainty, dependence, model complexity, and confidence into explicit terms, providing a unified route from CV scores to risk statements and a principled validation criterion for heterogeneous small-sample applications such as neuroimaging.