Search papers, labs, and topics across Lattice.
This paper introduces a method to select an optimal, fixed momentum parameter $\beta$ for Adam by training a local model of Adam's normalized direction on a short pilot run. This model balances sampling variability against gradient averaging delay, yielding a cubic memory rule whose coefficients are estimated from gradient probes. Retrospective evaluations on vision and language tasks demonstrate a significant reduction in validation gap compared to standard fixed $\beta$ or grid search approaches.
A cubic memory rule for Adam, derived from a local model of gradient dynamics, slashes validation gaps by over 40% on diverse vision and language benchmarks.
We propose a method for choosing the shared memory parameter $\beta_1=\beta_2=\beta$ in Adam from a short pilot training. The selected $\beta$ remains fixed during the subsequent full training. A local model of Adam's normalized direction balances sampling variability against the delay introduced by averaging past gradients. This balance gives a cubic memory rule, whose two coefficients are estimated from gradient probes at a few pilot checkpoints. The estimator uses the numerator and denominator jointly, preserving their covariance. With a 200-update pilot and sixteen probe gradients at each of four checkpoints, a seed-matched retrospective evaluation on eleven vision and language workloads reduces mean relative validation gap by 40.7% and worst-quarter mean gap by 44.3% against the grid representative of shared $\beta=0.95$. The mean gap is also 32.3% lower than that of the best constant $\beta$ chosen across all eleven workloads.
