Search papers, labs, and topics across Lattice.
This paper investigates the memorization capacity of deep ReLU networks, focusing on the minimal width and depth required to memorize $N$ data points with pairwise separation $δ$. The authors construct networks with width $W$ and depth $L$ satisfying $W^2L^2= \mathcal{O}(N\log(δ^{-1}))$ that can memorize any $N$ data samples. They also prove a matching lower bound of $W^2L^2=Ω(N \log(δ^{-1}))$, establishing an optimal trade-off between width and depth for memorization capacity, up to logarithmic factors.
Forget parameter counts – the true memorization capacity of deep ReLU networks is fundamentally bounded by the product of squared width and squared depth, $W^2L^2$, scaling linearly with data size.
This paper studies the memorization capacity of deep neural networks with ReLU activation. Specifically, we investigate the minimal size of such networks to memorize any $N$ data points in the unit ball with pairwise separation distance $δ$ and discrete labels. Most prior studies characterize the memorization capacity by the number of parameters or neurons. We generalize these results by constructing neural networks, whose width $W$ and depth $L$ satisfy $W^2L^2= \mathcal{O}(N\log(δ^{-1}))$, that can memorize any $N$ data samples. We also prove that any such networks should also satisfy the lower bound $W^2L^2=Ω(N \log(δ^{-1}))$, which implies that our construction is optimal up to logarithmic factors when $δ^{-1}$ is polynomial in $N$. Hence, we explicitly characterize the trade-off between width and depth for the memorization capacity of deep neural networks in this regime.
