Search papers, labs, and topics across Lattice.
This paper introduces a unified statistical framework for hyperparameter selection that shifts from empirical methods to a learn-then-test (LTT) paradigm, allowing for formal guarantees on reliability and safety. By framing hyperparameter selection as multiple hypothesis testing, the authors demonstrate how to select parameters that meet specific reliability requirements with controlled error probabilities. Key results include the derivation of statistical tools such as p-values and e-values that underpin the framework, ensuring finite-sample performance in real-world applications.
Hyperparameter selection can now be backed by formal statistical guarantees, transforming a traditionally heuristic process into a reliable, principled approach.
Hyperparameter selection is a critical step in the deployment of modern artificial intelligence systems, given the need to tune degrees of freedom such as inference-time parameters, implementation-level settings, and thresholds driving decision rules. Despite its practical importance, hyperparameter selection is typically performed using best-effort empirical methods such as grid search or Bayesian optimization, which provide no formal statistical guarantees on reliability or safety. This monograph presents a unified statistical framework for reliable hyperparameter selection, centered on the learn-then-test (LTT) paradigm, which formulates the problem as multiple hypothesis testing over a candidate set of hyperparameters. The framework enables the selection of hyperparameters that provably satisfy application-specific reliability requirements -- such as bounds on average risk, quantile risk, or information-theoretic constraints -- with explicit, finite-sample control of error probabilities. The supporting statistical machinery, namely p-values, e-values, and concentration inequalities, is developed from first principles in a dedicated appendix.
