Search papers, labs, and topics across Lattice.
This paper explores a gradient-free training method for deep neural networks using a simple Monte Carlo algorithm that randomly mutates parameters and retains those that reduce loss. The approach circumvents the issues of vanishing and exploding gradients associated with backpropagation, demonstrating its effectiveness across various architectures, including deep networks with over 20 layers and Transformers. Notably, this method allows for training without batch normalization or residual connections, revealing significant redundancy in deep networks and offering new insights into their learning mechanisms.
A simple Monte Carlo method can effectively train deep neural networks without relying on gradients, revealing surprising redundancies in their architectures.
Backpropagation (BP) dominates deep learning training, but its reliance on gradients brings inherent troubles -- vanishing and exploding gradients. The pursuit of gradient-free methods has long been a goal in the field of artificial intelligence. This paper shows that indeed the simplest Monte Carlo algorithm implemented on a single GPU -- randomly mutate a parameter, keep it if the loss decreases, otherwise retry -- can practically train deep networks. This gradient-free method does not even need common techniques such as batch normalization or residual connections to directly train sufficiently deep networks. More remarkably, its flexibility extends to several nontrivial scenarios: it enables pure pruning training, supports discrete weights, accommodates unconventional transfer functions such as Gaussian, and reveals the substantial redundancy of deep networks. We have demonstrated its feasibility on deep networks with more than 20 layers, single-hidden-layer wide networks with up to 16,384 hidden neurons, and even a simple Transformer architecture trained on both image classification (MNIST) and character-level language modeling (Tiny Shakespeare). This simple gradient-free method may offer a complementary perspective for understanding the self-organization and learning mechanisms of neural networks, and also provides an alternative route for building physically inspired deep learning systems.
