Search papers, labs, and topics across Lattice.
This study explores the effectiveness of different activation functions in neural networks tasked with learning the dynamics of Conway's Game of Life, revealing that networks with polynomial activation functions significantly outperform those using standard Rectified Linear Units. By reframing the learning challenge as one that benefits from tailored inductive biases rather than sheer network size, the authors demonstrate that even minimal networks can successfully capture complex dynamics. The findings advocate for a shift in approach towards using simpler, more appropriate models for specific tasks, potentially transforming strategies in machine learning applications across various scientific domains.
Switching to polynomial activation functions allows minimal neural networks to master Game of Life dynamics, challenging the notion that bigger is always better in neural network design.
Previous work has found a gap between the scale of neural networks that reliably learn Conway's Game of Life, and minimal networks capable of representing the classic cellular automaton with hard-coded parameter values. Viewing neural network learning as a search process suggests a dependence on networks large enough to contain sub-networks with lucky initializations (sometimes known as 'winning tickets') that actually learn the task. In this work, we reorient our perspective from discovering Life rules as a search problem back to a learning problem, and reason that with fitting inductive biases, the problem should be much more amenable to minimal networks. We find that network variants with several alternative activation functions meaningfully outperform the default choice of Rectified Linear Units, and in particular, that a 2nd degree polynomial activation function consistently learns Life dynamics with or without the benefit of learning neural weights. Our results provide an informative demonstration of the benefits of matching learning to the task at hand and challenge the easy default choice of scale for all problems. In particular, we advocate for the use of cellular automata as simple test domains for developing strategies that can benefit machine learning for science, physics-based deep learning, and interpretable machine learning.
