Search papers, labs, and topics across Lattice.
This paper explores the limitations of learning $\tanh$ neural networks under finite precision, revealing that adaptive randomized algorithms cannot exceed a convergence rate of $O(m^{-1/p})$ in the $L^p$ norm without an exponentially increasing sampling budget relative to network size. The authors utilize a novel construction of sharply localized bump functions through iterated $\tanh$ activations to substantiate their findings. These results highlight significant constraints on the learnability of localized bump function classes in finite precision, extending previous insights from ReLU networks to the $\tanh$ framework.
Adaptive algorithms for $\tanh$ networks hit a hard ceiling in convergence rates due to finite precision, challenging assumptions about their learnability.
We investigate limitations of learning $\tanh$ neural networks from point evaluations under finite-precision computations and $L^p$ accuracy guarantees, building on Berner, Grohs, and Voigtl盲nder (2023). Our approach is based on a novel construction of sharply localized bump functions via iterated $\tanh$ activations. Using this mechanism, we show that, in a finite-precision setting, no adaptive randomized algorithm based on $m$ samples can achieve a convergence rate higher than the Monte Carlo rate $O(m^{-1/p})$ in the $L^p$ norm, unless the sampling budget grows exponentially with the size of the network parameters and architecture. The results reveal fundamental limitations imposed by finite precision on the learnability of classes containing localized bump functions, extending previous results for ReLU networks to the $\tanh$ setting.
