Track genomics, neuroscience and molecular biology.
Search papers, labs, and topics across Lattice.
This paper introduces IntWorld and PermWorld, procedurally generated mathematical task ecosystems designed to isolate the effect of task diversity on neural network representations. By varying the number of related but distinct tasks within these controlled environments, the authors investigate how task variety influences model internal states. Initial results suggest that increased task diversity leads to richer internal representations, offering a controlled approach to studying emergent capabilities.
Task diversity, not just task complexity, significantly enhances the richness of neural network internal representations in controlled mathematical environments.
Basic mathematical tasks like modular addition have been a valuable tool for illuminating aspects of neural network behavior that are challenging to see in noisy, unverifiable, large-scale, real-world settings. These studies have typically focused on a single task. However, modern language models are exposed to a vast number of tasks despite being trained with a next-token prediction objective. In this work, we introduce two mathematical task ecosystems, IntWorld and PermWorld, built from distinct but related tasks on integers and permutations. Both are designed to simulate task diversity within a cohesive mathematical'world'where tasks are distinct but related. Because the datasets are procedurally generated, we can precisely vary the number of tasks, allowing us to begin to disentangle the role of task variety from other factors in training. After describing the datasets, we provide some initial results obtained from these datasets which support the idea that increasing task diversity increases the richness of a model's internal representations.