Search papers, labs, and topics across Lattice.
This paper introduces a hybrid least squares/gradient descent (LSGD) method tailored for MIONets, significantly enhancing training efficiency. By treating MIONets as multilinear functions, the authors optimize the last layer parameters of branch networks using an alternating least squares approach, which simplifies the computation of large system matrices through Kronecker and Khatri-Rao products. The key result demonstrates that this method can effectively handle $L^2$ loss with regularization, leading to faster convergence in training MIONets compared to traditional methods.
Accelerating MIONet training with a novel hybrid LSGD method could redefine efficiency benchmarks in deep learning architectures.
In this paper, we propose an efficient hybrid least squares/gradient descent (LSGD) method for MIONets to accelerate training. This method generalizes the LSGD method for DeepONets. Since MIONet is the sum of the entrywise product of multiple branch networks and a trunk network, it can be viewed as a multilinear function with respect to the last layer parameters of each branch network. These sets of parameters can be optimized using the alternating least squares method, where we solve the LS system for a single branch network in turn. To handle the large-sized system matrix, we introduce Kronecker and Khatri-Rao products and tensor permutation matrices to factor the large matrix into small ones. Our method is compatible with a general type of $L^2$ loss with regularization terms for the last layer parameters of each branch, where linear operators can be applied to the MIONet output in each loss term.
