Track genomics, neuroscience and molecular biology.
Search papers, labs, and topics across Lattice.
We present the Caterpillar Net, a non-recursive two-hidden-layer construction capable of representing any continuous piecewise linear (CPWL) function as a finite signed sum of maxima of affine functions. This construction addresses the expressiveness of ReLU networks, providing a concrete solution to the problem of representing general CPWL functions with limited depth. The network achieves this representation with widths of $O(N^2)$ and $2^{O(N\log N)}$, offering a direct geometric interpretation of the function's structure.
General CPWL functions can be represented by ReLU networks with just two hidden layers, shattering previous depth conjectures.
Every continuous piecewise linear (CPWL) function is a finite signed sum of maxima of affine functions, making maximum representation a central problem in the study of the expressiveness of ReLU networks. A now-disproved conjecture predicted logarithmic depth for maxima. Inspired by the proof by Brunck et al. (2026) that two hidden layers suffice for general CPWL functions, we give the Caterpillar Net, a nonrecursive construction for the maximum of $N$ inputs with two hidden layers of widths $O(N^2)$ and $2^{O(N\log N)}$. The construction has a direct geometric interpretation.