Search papers, labs, and topics across Lattice.
This paper introduces HyperMC, a multi-fidelity hyperparameter tuning framework specifically designed for stochastic gradient Markov chain Monte Carlo (SGMCMC) methods, which are critical for scalable Bayesian inference. By integrating Hyperband-style resource allocation with kernel Stein discrepancy (KSD) evaluation, HyperMC effectively explores the hyperparameter space while ensuring accurate evaluations within a fixed computational budget. Experimental results demonstrate that HyperMC outperforms existing methods like MAMBA and grid search in terms of posterior approximation and predictive calibration, with Robust HyperMC providing enhanced stability and reproducibility in tuning outcomes.
HyperMC achieves superior hyperparameter tuning for SGMCMC, significantly improving Bayesian inference performance while ensuring reproducibility.
Stochastic gradient Markov chain Monte Carlo (SGMCMC) methods enable scalable Bayesian inference, but their performance depends strongly on hyperparameters such as the step size, mini-batch size, and number of leapfrog steps. Since most SGMCMC algorithms lack a Metropolis-Hastings acceptance rate, standard acceptance-based tuning methods are not directly applicable. We propose HyperMC, a multi-fidelity tuning framework that combines Hyperband-style resource allocation with kernel Stein discrepancy (KSD) evaluation. By running multiple successive-halving brackets, HyperMC balances broad exploration of a continuous hyperparameter space with increasingly accurate evaluation of promising configurations under a fixed computational budget. We further introduce Robust HyperMC, which uses global grid initialization followed by elite-guided local refinement to reduce sensitivity to random candidate generation and noisy finite-budget evaluations. Under suitable approximation and concentration conditions for the estimated KSD, we establish that the successive-halving component selects a near-optimal configuration among the sampled candidates with high probability and derive a sufficient computational budget for successful selection. Experiments on logistic regression, probabilistic matrix factorization, and Bayesian neural networks show that HyperMC improves posterior approximation or predictive calibration relative to MAMBA, grid search, and heuristic baselines, while Robust HyperMC yields more stable and reproducible tuning results.
