Search papers, labs, and topics across Lattice.
This paper introduces a unified function space theory for deep fully connected neural networks, defining functions recursively with $\ell^1$-bounded linear combinations of activated functions. The framework accommodates a wide range of activation functions, both homogeneous and non-homogeneous, and reveals that depth does not necessarily enhance functional diversity when complexity is norm-controlled. Key findings include a "depth saturation" result for ReLU networks, indicating that increased depth leads to only minor rescaling of function classes without additional expressivity benefits.
Depth in neural networks may not enhance expressivity as previously thought, revealing that functional diversity is constrained by norm-controlled complexity.
We develop a unified function space theory of deep fully connected neural networks. Functions in our spaces are defined recursively as $\ell^1$-bounded linear combinations of activated functions from preceding layers, with a dictionary of affine functions at the first layer. Unlike existing theories that are largely specialized to homogeneous activations such as the ReLU, our framework provides a meaningful notion of functional complexity for deep networks with a broad range of homogeneous and non-homogeneous activation functions commonly used in practice. This simple construction unites several seemingly disparate ideas from the literature, including norm-based complexity bounds and variational characterizations of depth, and facilitates novel analyses of what kinds of functions deep norm-constrained networks can represent. To this end, we prove a novel representer theorem for our spaces and establish novel function-space complexity bounds showing that the associated function classes remain qualitatively small at arbitrary depth. In the univariate ReLU case, we prove a "depth saturation" result: depth in this setting yields only a small constant rescaling of the function class, with no added functional diversity. As a consequence, we show that deep norm-controlled ReLU functions in any dimension cannot exhibit high frequencies along any direction. This finding reveals that some commonly cited expressivity benefits of depth disappear once network complexity is controlled by an appropriate function space norm, rather than parameter count or other representational costs that permit compounded rescaling across layers. Overall, our results illustrate how a function space perspective yields new structural insights into the relationship between depth and complexity.
