Search papers, labs, and topics across Lattice.
This study investigates the delayed influence of minibatch perturbations in the AdamW optimizer by modeling it as a finite-horizon input-state-output (ISO) system. By linearizing the dynamics, the authors derive a signed response operator that quantifies how localized gradient updates affect future loss, revealing the optimizer's memory effects on the magnitude, timing, and sign of these influences. Experimental validation confirms the proposed response mechanism and highlights the significant prospective structure in delayed effects, which can be partially approximated using ISO methods.
Minibatch perturbations in AdamW can have delayed and significant impacts on future loss, reshaping our understanding of optimizer dynamics.
A minibatch can influence training beyond the update at which it is observed because AdamW stores past gradient information in its optimizer states. We study this delayed effect through paired trajectories that differ only in one gradient update and share the same subsequent training sequence. We formulate AdamW as a finite-horizon input--state--output (ISO) system whose state contains the model parameters and first- and second-moment estimates. Linearizing the joint dynamics yields a signed response operator that maps a localized gradient perturbation to its future loss effects, revealing how optimizer memory shapes their magnitude, timing, and sign. We further derive an exact multistep error decomposition and establish first-order finite-horizon accuracy under local smoothness and controlled activation switching. Experiments validate the response mechanism and optimizer-state effects, while repeated-future analyses reveal substantial prospective structure in delayed influence that can be partially recovered from ISO approximations. Code is available at https://github.com/Kanyooo/Loss_ISO.
