Search papers, labs, and topics across Lattice.
This paper introduces LITTLECURRICULUM, a meticulously curated 88B-token pretraining corpus focused on U.S. elementary school material, which excludes content beyond Grade 5 to facilitate the study of knowledge and skill acquisition in language models. Training a 5B-parameter LLM on this dataset results in LITTLELEARNER, a model that demonstrates language competence while maintaining clear boundaries in knowledge and capabilities aligned with educational guidelines. The research highlights the effectiveness of this controlled training environment for exploring knowledge injection methods, revealing that while LITTLELEARNER can better utilize existing knowledge, it does not extend its capabilities beyond the predefined scope.
LITTLELEARNER reveals that even a well-defined knowledge scope can yield a competent language model, but it won't expand its capabilities beyond its educational boundaries.
Modern language models are trained on heterogeneous web-scale text corpora. Consequently, studying knowledge and skill acquisition is difficult, as prior exposure to related content is hard to characterize. To address this challenge, we introduce LITTLECURRICULUM, a curated 88B-token pretraining corpus tailored to U.S. elementary school material, explicitly excluding concepts, facts, and vocabulary taught above Grade 5. Training a 5B-parameter LLM from scratch on LITTLECURRICULUM yields LITTLELEARNER, a model with sufficient language competence for open-ended evaluation, yet with clear knowledge and capability boundaries mapped to interpretable curriculum guidelines. We release LITTLECURRICULUM and LITTLELEARNER as a developmentally restricted sandbox to study how models acquire, represent, and use data under a well-defined training scope. We illustrate the sandbox's utility in a first suite of experiments on injecting new knowledge through post-training and in-context learning. These methods let LITTLELEARNER better utilize existing knowledge, but do not raise out-of-scope capabilities. Our findings underscore the value of this controlled environment for future investigations.
