Search papers, labs, and topics across Lattice.
This paper develops a Landau-type theory to bridge the gap between invariant learning objectives and their behavioral outcomes, framing representation learning as a multimode magnetization process. By deriving an effective free energy from invariant-learning objectives, the authors identify distinct regularization phenotypes and critical strengths that influence model behavior during training. Controlled experiments validate the theoretical predictions, revealing that low-order objective signatures can effectively predict how models learn as regularization strength changes.
Understanding how regularization influences model learning could revolutionize our approach to designing robust AI systems.
Invariant learning seeks representations that remain predictive across environments, yet the behavior of its objectives along the regularization path is often opaque. We address this objective-behavior gap by viewing representation learning as multimode magnetization and deriving, from concrete invariant-learning objectives, a Landau-type effective free energy whose low-order coefficients form objective signatures and induce distinct regularization phenotypes. Effective quadratic corrections move the phase boundary and enable finite-strength mode elimination; quartic corrections regulate post-onset amplitude and typically leave residual loading at finite strength; higher-order structure governs non-monotone tails, instability, and collapse at large regularization. In a canonical bilinear model, the theory yields closed-form phase boundaries and steady-state loadings, as well as distinct critical strengths for shortcut and stable modes that define a selective-retention window. Controlled experiments confirm the predicted phase boundaries, loadings, and regularization phenotypes. In one- and two-hidden-layer ReLU networks, the same signatures remain predictive of qualitative regularization-path behavior despite depth-dependent shifts in scale. A matrix extension generalizes the framework to coupled collective modes and yields a spectral phase-boundary criterion. Together, the framework turns low-order objective signatures into predictions of regularization phenotypes and, ultimately, of what models learn as regularization varies.
Tsinghua AI