Track synthesis, materials and catalysis as it is published.
Search papers, labs, and topics across Lattice.
This paper investigates how pre-training knowledge organization impacts the efficiency of knowledge updates in language models, finding that chronological ordering leads to "specialist" models adept at recent information. These specialists exhibit significantly faster knowledge acquisition for new data compared to "generalist" models trained on shuffled data. A derived theoretical law explains this update efficiency advantage by linking it to optimization dynamics, guiding a data organization strategy that balances stable historical knowledge with efficient current updates.
Chronologically ordered pre-training creates "specialist" models that learn new knowledge orders of magnitude faster than generalists.
An ideal (language) model must continually update its knowledge. Existing work largely focuses on making these updates efficient after training. We investigate how the organization of time-varying knowledge during pre-training shapes the efficiency of subsequent updates. Models trained on chronologically ordered knowledge specialize in the most recently encountered knowledge; we call them specialists. In contrast, models trained on the same knowledge in randomly shuffled order perform well across all represented time periods; we call them generalists. Surprisingly, specialists require far fewer update steps than generalists to acquire new knowledge. To explain this advantage, we derive a law from first principles that relates update cost to the model's optimization dynamics and that closely predicts empirical results. Guided by these findings, we design a strategy for organizing pre-training data that preserves stable historical knowledge while enabling efficient updates to current knowledge. We also show that the benefits of temporal knowledge organization extend to real-world pre-trained models and existing knowledge update tools, and generalize to other knowledge orderings. Project page: https://horizon-lab.org/specialist/