Search papers, labs, and topics across Lattice.
This paper introduces a multilevel training algorithm for Kolmogorov-Arnold Networks (KANs) that exploits their structured function compositions and spline basis expansions. They establish an equivalence between KANs and multichannel MLPs via a linear change of basis, which motivates a training approach involving a sequence of KANs with uniformly refined spline knots and analytic geometric interpolation. Experiments show orders of magnitude improvement in accuracy over conventional methods for training KANs or MLPs, especially for physics-informed neural networks.
KANs get a huge training boost: multilevel training achieves orders of magnitude accuracy gains over standard methods, especially for physics-informed neural networks.
Algorithmic speedup of training common neural architectures is made difficult by the lack of structure guaranteed by the function compositions inherent to such networks. In contrast to multilayer perceptrons (MLPs), Kolmogorov-Arnold networks (KANs) provide more structure by expanding learned activations in a specified basis. This paper exploits this structure to develop practical algorithms and theoretical insights, yielding training speedup via multilevel training for KANs. To do so, we first establish an equivalence between KANs with spline basis functions and multichannel MLPs with power ReLU activations through a linear change of basis. We then analyze how this change of basis affects the geometry of gradient-based optimization with respect to spline knots. The KANs change-of-basis motivates a multilevel training approach, where we train a sequence of KANs naturally defined through a uniform refinement of spline knots with analytic geometric interpolation operators between models. The interpolation scheme enables a ``properly nested hierarchy''of architectures, ensuring that interpolation to a fine model preserves the progress made on coarse models, while the compact support of spline basis functions ensures complementary optimization on subsequent levels. Numerical experiments demonstrate that our multilevel training approach can achieve orders of magnitude improvement in accuracy over conventional methods to train comparable KANs or MLPs, particularly for physics informed neural networks. Finally, this work demonstrates how principled design of neural networks can lead to exploitable structure, and in this case, multilevel algorithms that can dramatically improve training performance.
