Search papers, labs, and topics across Lattice.
This paper investigates the relationship between learned symmetries in neural networks and their parameter space, formulating a lifting problem to understand how symmetries can be represented through parameter adjustments. By analyzing the functional sensitivities of individual parameters, the authors establish conditions under which a smooth parameter-space action corresponds to the symmetry in function space. Key findings reveal that while certain parameter directions can effectively track symmetry orbits locally, fixed directions diverge from these orbits after training, highlighting the limitations of parameterization in capturing learned symmetries.
Symmetries in neural networks can be tracked through parameter adjustments, but fixed directions fail to maintain this relationship post-training.
When a network has learned a function with a known symmetry, can that symmetry be moved through the parametrisation---is there a motion in parameter space realising the group action in function space? We formulate this as a lifting problem for the realisation map $Φ:θ\mapsto f_θ$, and show that a smooth parameter-space action exists only if the tangent space to the function's symmetry orbit lies within the image of $\mathrm dΦ_θ$, whose columns are the \emph{functional sensitivities} of individual parameters. This condition is also sufficient for pointwise first-order lifting. Relaxing it in least squares yields two local parameter directions: one following the symmetry orbit, one descending towards the equivariant subspace, with residuals measuring what the parametrisation cannot reach. On a rotationally invariant classifier we find these directions induce their predicted function-space motion, but only locally: recomputed directions track the orbit and reduce the equivariance defect, while directions held fixed depart from both after training. The same holds for Hamiltonian neural networks trained on a rotationally symmetric potential, even though the architecture does not explicitly enforce the symmetry.
