Search papers, labs, and topics across Lattice.
This paper explores PAC-Bayes derandomization for smooth loss functions, aiming to derive generalization bounds for deterministic predictors by leveraging the smoothness of loss and predictor classes. The authors establish that transitioning from Gibbs predictors to deterministic ones incurs a specific cost linked to the Jensen gap class's generalization gap, which they control using Rademacher complexity. Their findings yield bounds that incorporate flatness metrics derived from parameter Jacobians and Hessians, and they validate a practical regularizer for BatchNorm networks through experiments on CIFAR-10, demonstrating its efficacy across varying batch sizes.
Transitioning from probabilistic to deterministic predictors incurs a quantifiable cost that can be precisely controlled, revealing new insights into generalization bounds.
We study PAC-Bayes derandomization for smooth loss functions. Our goal is to obtain generalization bounds that hold with high probability for deterministic predictors by exploiting smoothness properties of both the loss and the predictor class. We show that passing from the Gibbs predictor to the deterministic predictor at the posterior mean has a precise cost, given by the generalization gap of the Jensen gap class. We control this class through its Rademacher complexity, leading to bounds for deterministic predictors that involve flatness quantities expressed in terms of parameter Jacobians and Hessians of the score map. The framework applies to both bounded and unbounded smooth loss functions, and we specialize the results to linear predictors and smooth neural networks. Finally, the Jacobian and Hessian quantities appearing in the theory motivate a practical regularizer. For BatchNorm networks, we compute this regularizer with respect to effective BatchNorm weights obtained by folding the BatchNorm transformation into the adjacent affine weights. Experiments on CIFAR-10 illustrate the behavior of this regularizer under different batch sizes.
