Search papers, labs, and topics across Lattice.
This study explores the low-dimensional topology of deep neural networks by constraining each layer to a width of \(d = 3\) to analyze how topological invariants evolve through layers. The authors demonstrate that the linking number serves as a key metric, revealing that ResNets and transformers exhibit comparable expressivity due to their layer-skipping and attention mechanisms, respectively, while traditional feedforward networks lag behind unless nonmonotonic activations are employed. These findings underscore the potential of low-dimensional topology as a framework for understanding and designing neural network architectures.
ResNets and transformers are equally powerful in altering linking numbers, challenging conventional views on network expressivity.
We study layered models, including feedforward networks, ResNets, and transformers, by limiting each layer to a width of $d = 3$, i.e., $\mathbb{R}^3$ as representation space. This allows us to track how a neural network changes low-dimensional topological invariants through its layers. Just about any topological structure may be simplified or even trivialized by simply increasing dimension; e.g., any knot is equivalent to an unknot in $\mathbb{R}^4$. By restricting to $\mathbb{R}^3$, we not only isolate the effects of activation and depth from that of width, we work in a space that lends itself to easy visualization. We focus on linking number here, deferring other invariants like link groups, Milnor's $\bar渭$-invariants, knot types, ambient cobordisms, to a sequel. We provide full proofs and empirical experiments to justify the following insights: When measured by their power to effect changes in linking numbers, the layer-skipping feature in ResNets is as powerful as the attention mechanism in transformers; both ResNets and transformers are strictly more powerful than feedforward neural networks with monotonic activations, which are in turn more powerful than invertible and flow-based models; but replacing monotonic activation with a nonmonotonic one elevates a feedforward network into the same expressivity class as ResNets and transformers. These results suggest that low-dimensional topology can be a useful tool to guide designs of AI architectures. We also generalize our results from $d = 3$ to arbitrary $d > 3$.
