Search papers, labs, and topics across Lattice.
This paper introduces an inertial modification to Dirac-Frenkel dynamics to address the challenges of non-unique or ill-conditioned parameter dynamics in nonlinear parametrizations like neural networks. By incorporating inertia, the method retains useful past velocity information while ensuring that well-informed directions adhere to the original dynamics, resulting in well-posed parameter evolution. Numerical experiments validate the robustness of this approach, showing improved performance in managing parameter dynamics compared to standard methods.
Inertia in Dirac-Frenkel dynamics significantly enhances parameter evolution stability, making it a game-changer for training complex models like neural networks.
Even when Dirac-Frenkel dynamics determine a well-defined evolution in function space, the corresponding parameter dynamics can be non-unique or ill-conditioned for redundant nonlinear parametrizations such as neural networks or mixture models. We propose to add inertia to the Dirac-Frenkel dynamics and show that this allows useful parameter velocity information to persist from the past trajectory in directions that are weakly informed, while well-informed parameter velocity directions continue to follow the Dirac-Frenkel dynamics. We prove that the inertial formulation yields well-posed parameter dynamics and provide a posteriori error bounds. After time discretization, the method requires the solution of the same type of regularized linear least-squares problem as standard Dirac-Frenkel dynamics, but with the previous velocity appearing as an anchor. Numerical experiments demonstrate the increased robustness obtained with inertia.
