Search papers, labs, and topics across Lattice.
This paper investigates the continuous reparametrization symmetry in neural networks with positively homogeneous activations, interpreting it as a gauge redundancy. The authors introduce gauge-adapted coordinates to separate invariant and scale-imbalance directions, and propose a soft orbit-selection functional to penalize scale imbalance. Experiments demonstrate that this penalty expands the stable learning-rate regime and suppresses scale drift, linking gauge-orbit geometry and optimization conditioning.
Soft gauge fixing stabilizes training and expands the learning rate regime for homogeneous neural networks by penalizing redundant scale imbalances.
Neural networks with positively homogeneous activations exhibit an exact continuous reparametrization symmetry: neuron-wise rescalings generate parameter-space orbits along which the input--output function is invariant. We interpret this symmetry as a gauge redundancy and introduce gauge-adapted coordinates that separate invariant and scale-imbalance directions. Inspired by gauge fixing in field theory, we introduce a soft orbit-selection (norm-balancing) functional acting only on redundant scale coordinates. We show analytically that it induces dissipative relaxation of imbalance modes to preserve the realized function. In controlled experiments, this orbit-selection penalty expands the stable learning-rate regime and suppresses scale drift without changing expressivity. These results establish a structural link between gauge-orbit geometry and optimization conditioning, providing a concrete connection between gauge-theoretic concepts and machine learning.
