Search papers, labs, and topics across Lattice.
This study investigates whether language models can independently discover the mathematical concept of zero, a critical test of their out-of-distribution generalization capabilities. The authors find that GPT-2-sized models fail to generalize this concept at test time without prior exposure, but can significantly improve their performance after being trained on numerous examples of zero. Furthermore, they demonstrate that language pretraining can halve the number of examples needed for successful discovery, suggesting that language skills enhance mathematical reasoning in neural networks.
Language models struggle to discover fundamental mathematical concepts like zero without extensive training, but language pretraining can dramatically reduce the learning burden.
AI systems based on artificial neural networks are being developed with aspirations of pushing the boundary of human mathematical knowledge. A key question for these systems is how much they can reach beyond their training data. Mathematical discovery requires a strong form of out of distribution generalization; the ability to hypothesize genuinely new - and potentially logically more powerful - mathematical structures. It has been hypothesized that language abilities support such generalizations in human cognition. In this work, we use simple arithmetic as a case study for examining how modern AI models could expand their mathematical horizons, evaluating whether these models can independently discover the concept of "zero". We show that We show that (1) language models of a GPT-2 size are unable to perform this generalization at test time regardless of language pretraining, but (2) models can improve substantially after training on tens or hundreds of examples of zero. Additionally, we find that language pretraining reduces the number of required examples by approximately $50\%$, showing that language abilities can scaffold mathematical discovery in neural models.
