Search papers, labs, and topics across Lattice.
This paper presents a modular framework for deriving high-probability regret bounds for empirical risk minimization (ERM) by outlining a three-step recipe involving a basic inequality, a uniform local concentration bound, and a fixed-point argument. It establishes regret bounds in terms of a critical radius, defined via localized Rademacher complexity, under a Bernstein-type variance-risk condition and demonstrates how to upper bound this radius for common function classes. The guide also reviews ERM with nuisance components, particularly in the context of causal inference and domain adaptation, providing regret-transfer bounds and analyzing both sample-splitting and in-sample regimes.
Unlock sharper regret bounds for ERM by following this guide's modular approach, which distills rate derivations into a three-step recipe and provides tools for handling nuisance components.
This guide develops high-probability regret bounds for empirical risk minimization (ERM). The presentation is modular: we state broadly applicable guarantees under high-level conditions and give tools for verifying them for specific losses and function classes. We emphasize that many ERM rate derivations can be organized around a three-step recipe -- a basic inequality, a uniform local concentration bound, and a fixed-point argument -- which yields regret bounds in terms of a critical radius, defined via localized Rademacher complexity, under a mild Bernstein-type variance--risk condition. To make these bounds concrete, we upper bound the critical radius using local maximal inequalities and metric-entropy integrals, recovering familiar rates for VC-subgraph, Sobolev/Hölder, and bounded-variation classes. We also review ERM with nuisance components -- including weighted ERM and Neyman-orthogonal losses -- as they arise in causal inference, missing data, and domain adaptation. Following the orthogonal learning framework, we highlight that these problems often admit regret-transfer bounds linking regret under an estimated loss to population regret under the target loss. These bounds typically decompose regret into (i) statistical error under the estimated (optimized) loss and (ii) approximation error due to nuisance estimation. Under sample splitting or cross-fitting, the first term can be controlled using standard fixed-loss ERM regret bounds, while the second term depends only on nuisance-estimation accuracy. We also treat the in-sample regime, where nuisances and the ERM are fit on the same data, deriving regret bounds and giving sufficient conditions for fast rates.
