Search papers, labs, and topics across Lattice.
This paper introduces a unified framework that connects information theory, topology, and statistical mechanics to address the question of why overparameterized deep networks generalize effectively despite theoretical predictions of overfitting. The authors establish the Entropic Learnability Horizon (ELH), a principle that delineates the conditions under which a neural network can learn a target function based on the interplay of Shannon entropy, topological entropy, and weight space entropy. Key findings include the Shannon-Topological Bottleneck Theorem, which reveals that exceeding the informational horizon leads to a state of Informational Frustration, where generalization is thermodynamically impossible, and the concept of "grokking" as an Entropic Release that allows networks to overcome this bottleneck.
When geometric complexity of decision boundaries exceeds a critical threshold, neural networks can enter a state of Informational Frustration, making generalization thermodynamically impossible.
Why overparameterised deep networks generalise so remarkably well remains one of the most stubborn open questions in machine learning theory. Classical frameworks like VC dimension and Rademacher complexity predict catastrophic overfitting in modern models, leaving a massive theoretical gap between theory and reality. In this paper, we bridge this divide by introducing a unified framework that links information theory, topology, and statistical mechanics to map the hard limits of deep learning. Central to our approach is the Entropic Learnability Horizon (ELH): a fundamental law stating that a network can only truly learn a target function if the Shannon entropy of the data manifold outpaces the topological entropy of the function's decision boundary, balanced by the von Neumann entropy of the network's weight space. We establish the Shannon-Topological Bottleneck Theorem, proving that when a target boundary's geometric complexity exceeds this informational horizon, the system undergoes a sudden entropic phase transition. It falls into a state of Informational Frustration - a glassy, rigid memorization phase where generalization becomes thermodynamically impossible. Using this lens, we show that the enigmatic phenomenon of "grokking" is actually an Entropic Release, where weights abruptly reorganise to unlock the bottleneck. Finally, we translate this theory into practice with Entropic Gradient Descent (EGD), an optimization algorithm that dynamically manages weight entropy to keep learning on track. Ultimately, this work repositions entropy not just as a tool for tracking uncertainty but as the fundamental physical currency that dictates whether a machine can learn.
