Search papers, labs, and topics across Lattice.
This paper introduces a novel deep network architecture with fixed width and arbitrary depth, where each intermediate layer provides an approximation to the target function. The approximation error of each layer is quantitatively controlled by the $L^p$ modulus of continuity of the target function at a geometrically decreasing scale $N^{-\ell}$, where $\ell$ is the layer index. This provides a scale-dependent interpretation of depth, showing how it facilitates progressive refinement of the approximation.
Depth in neural networks isn't just about the final output; this work shows how each intermediate layer can be a progressively refined approximation, with error explicitly tied to the layer's geometric scale.
Depth is widely viewed as a central contributor to the success of deep neural networks, whereas standard neural network approximation theory typically provides guarantees only for the final output and leaves the role of intermediate layers largely unclear. We address this gap by developing a quantitative framework in which depth admits a precise scale-dependent interpretation. Specifically, we design a single shared mixed-activation architecture of fixed width $2dN+d+2$ and any prescribed finite depth such that each intermediate readout $\Phi_\ell$ is itself an approximant to the target function $f$. For $f\in L^p([0,1]^d)$ with $p\in [1,\infty)$, the approximation error of $\Phi_\ell$ is controlled by $(2d+1)$ times the $L^p$ modulus of continuity at the geometric scale $N^{-\ell}$ for all $\ell$. The estimate reduces to the geometric rate $(2d+1)N^{-\ell}$ if $f$ is $1$-Lipschitz. Our network design is inspired by multigrade deep learning, where depth serves as a progressive refinement mechanism: each new correction targets residual information at a finer scale while the earlier correction terms remain part of the later readouts, yielding a nested architecture that supports adaptive refinement without redesigning the preceding network.
