Search papers, labs, and topics across Lattice.
This paper explores new activation functions to achieve arbitrary-accuracy Sobolev approximation using fixed-size neural networks. The authors demonstrate that any function in the Sobolev space \(W^{2,\infty}((a,b)^d)\) can be approximated with arbitrary accuracy in the \(W^{1,\infty}\)-norm using the Elementary Universal Activation Function (EUAF), and extend this result to \(W^{s,\infty}((a,b)^d)\) with the introduction of the Differentiable Universal Activation Function \((DUAF_{\infty})\). Key findings include explicit width and depth bounds for these networks, showing that even sigmoidal variants can achieve similar approximation guarantees.
Fixed-size neural networks can achieve arbitrary Sobolev approximation accuracy with new activation functions, challenging traditional limits on network size and complexity.
In this work, we investigate new activation functions for achieving arbitrary-accuracy Sobolev approximation by fixed-size neural networks. We first show that any function in $W^{2,\infty}((a,b)^d)$ can be approximated with arbitrary accuracy, measured in the $W^{1,\infty}$-norm, by a fixed-size neural network using the Elementary Universal Activation Function ($\mathrm{EUAF}$). To extend this result to $W^{s,\infty}((a,b)^d)$ for $s\in\mathbb{N}$, we introduce a smooth activation $\mathrm{DUAF}_{\infty}$ from the family of Differentiable Universal Activation Functions ($\mathrm{DUAF}_n$). We prove that any function in $W^{s,\infty}((a,b)^d)$ can be approximated with arbitrary accuracy in the $W^{s-1,\infty}$-norm by a fixed-size $\mathrm{DUAF}_{\infty}$-activated network. We further construct sigmoidal variants $\widetilde{\mathrm{DUAF}}_n$ and show that, for every $1\leq s\leq n$, fixed-size $\widetilde{\mathrm{DUAF}}_n$-activated networks still approximate any $f\in W^{s,\infty}((a,b)^d)$ with arbitrary accuracy in the $W^{s-1,\infty}$-norm. In all these results, the width and depth bounds are computed explicitly, and the proposed activations are elementary.
