Search papers, labs, and topics across Lattice.
This paper introduces an information allocation dynamics perspective to understand how optimizer biases form during neural network training. By analyzing the contributions of weight and bias updates to residual signals, the authors reveal that the relative allocation of training signals between these pathways can be manipulated through a continuous preconditioning exponent \(p\). The findings highlight that optimizer implicit bias is not only evident in final solutions but also significantly influences training dynamics and generalization behavior throughout the optimization process.
The way training signals are allocated between weight and bias pathways can fundamentally alter the optimization dynamics and generalization of neural networks.
Different optimizers have different update biases, but these biases are usually implicit. Existing studies mainly analyze or control such biases from the geometry of the final solution. However, how optimizer bias forms during training still lacks a clear internal mechanism. This paper proposes an information allocation dynamics perspective. It interprets optimizer implicit bias as the relative allocation of training signals between weight-like and bias-like parameter pathways. This allocation can be described and adjusted by a continuous preconditioning exponent \(p\). To characterize this mechanism, we first analyze the update contributions of weight and bias to the same residual signal in a minimal linear model. The weight correction term preserves input-dependent residual signals, while the bias correction term preserves the residual mean direction. They therefore correspond to different projection pathways of the residual signal. After substituting the preconditioned update into the residual update equation, the optimizer can change the relative strength of the weight correction term and the bias correction term through different preconditioning factors. Therefore, optimizer implicit bias is not only reflected in the final solution or the global training trajectory. It is also reflected in the relative write-in ratio of training signals across different parameter pathways. Overall, this paper moves the analysis of optimizer implicit bias from solution-space geometry to update dynamics during training. It reveals that the relative update allocation between weight and bias-like parameters is an important dynamical mechanism that affects parameter trajectories and generalization behavior.
