Search papers, labs, and topics across Lattice.
This paper introduces a model-assisted sampling framework that leverages survey sampling theory to improve stochastic gradient estimation in machine learning optimization. By incorporating auxiliary gradient-prediction models, the method reduces variance in gradient estimates while maintaining compatibility with existing optimizers, leading to enhanced efficiency and generalization. Empirical results demonstrate significant performance improvements in 71-86% of experiments, particularly with momentum-based optimizers like AdamW, achieving better generalization in fewer training epochs compared to traditional methods.
Auxiliary gradient-prediction models can drastically reduce stochastic gradient noise, leading to faster convergence and better generalization in deep learning.
This work addresses the problem of variance in stochastic gradient estimation for machine learning optimization. Deep learning relies on mini-batch methods such as stochastic gradient descent, which approximate full gradients but introduce noise, creating trade-offs between convergence stability, speed, and generalization. Existing methods, including variance reduction techniques (e.g., SVRG and SAG) and adaptive optimizers, aim to mitigate gradient noise but may introduce additional computational overhead. We propose a model-assisted sampling framework that interprets mini-batch gradients through survey sampling theory, treating the dataset as a fixed finite population and gradients as sample-based estimates. Our aim is to bridge machine learning optimization and survey sampling theory by combining their perspectives on sample-based estimation and variance reduction. By incorporating auxiliary gradient-prediction models, we construct more efficient gradient estimators, with uniform sampling arising as a special case when no auxiliary information is used. Our approach integrates easily with existing optimizers, improving efficiency without altering their dynamics. Empirical results on synthetic and six benchmark datasets show performance gains in 71-86% of the experiments, particularly for medium-sized input spaces in our benchmarks. Notably, with momentum-based optimizers such as AdamW, the proposed estimator achieves clearly better generalization in roughly half the training epochs compared to baseline estimator.
