Search papers, labs, and topics across Lattice.
This study investigates the effectiveness of continual pre-training (CPT) versus pre-training from scratch (PTS) for adapting language models (LMs) to software engineering (SE) texts. The findings reveal that CPT consistently outperforms PTS across various model families and sizes, yielding minimal domain adaptation gains while maintaining general-language understanding, while PTS incurs significant penalties in both areas. The research provides practical guidance for optimizing LMs for SE applications and includes the release of a new SE corpus and pre-trained models for further exploration.
Reusing existing language models for software engineering texts significantly outperforms training new domain-specific models from scratch, challenging assumptions about domain adaptation strategies.
Generalist and code-focused Language Models (LMs) are increasingly applied to software engineering (SE), yet whether they are optimized for understanding SE textual artifacts (e.g., issues, commit messages, developer discussions) remains unclear, as most evidence comes from code-focused benchmarks. We study how to adapt encoder and decoder LMs to SE text, comparing continual pre-training (CPT) against pre-training from scratch (PTS) on a new SE corpus, and evaluating both domain adaptation (SELU) and general-language understanding (SuperGLUE). To keep the comparisons fair, we control pre-training under constant-token and compute-matched budgets. We find that across families and sizes, reusing an existing LM dominates training a domain-native one from scratch: CPT yields small and mostly inconclusive domain gains while leaving general-language understanding essentially unchanged, whereas PTS pays a large and usually decisive penalty on both axes and becomes competitive only for small LMs under a token-rich budget. We distill these results into practical guidance for adapting LMs to SE text and release our corpus and pre-trained LMs in our replication kit.
