Search papers, labs, and topics across Lattice.
This paper introduces Skill Pre-Training (SPT), a method that leverages publicly available skill packages as pre-training data for agentic language models, enhancing their tool-using capabilities. By employing a reference-aware assembly strategy called Reference Insert, SPT maintains the relationships among files within skill packages, ensuring effective integration during training. Experimental results demonstrate that SPT significantly boosts agentic performance compared to traditional mid-training approaches, while also preserving general performance metrics across various model scales.
Skill packages can transform agentic language models, improving their performance by utilizing reusable tool semantics and workflows during pre-training.
Agentic (tool-using) language models are mainly trained on tool-call traces and agent trajectories during post-training. These data provide direct behavioral supervision, but producing them requires task environments, execution, and verification, making broad tool and task coverage expensive. Publicly available skills offer another source of training data: they encode reusable tool semantics and workflows but are typically used only as inference-time context. We introduce Skill Pre-Training (SPT), a mid-training method that applies causal language modeling to SkillCorpus, a collection of public multi-file skill packages, optionally mixed with general data. To preserve relations among files within each package, we also introduce Reference Insert, a reference-aware assembly strategy that places supporting files near their mentions in the primary instruction. Experiments across multiple model scales and post-training recipes show that SPT consistently improves agentic performance over mid-training on general or trajectory data, while largely preserving general performance. Data mixture experiments show additional benefits from combining skill data with general annealing corpora. These results indicate that skill packages are a valuable data source for pre-training agentic language models.
