Search papers, labs, and topics across Lattice.
To resolve how best to adapt encoders to heterogeneous domain data, the authors evaluate continued pretraining of ModernBERT-Base across scientific literature, filtered web data, and synthetic text, directly comparing joint mixture training against post-hoc parameter-space merging. Understanding optimal corpus composition and integration strategies is critical for preventing negative transfer across highly disparate domain distributions. Across nine climate benchmarks, parameter-space merging consistently outperformed joint multi-source pretraining, yielding an average 76.3 F1 (+2.8 over vanilla ModernBERT) with academic text providing the dominant adaptation signal.
Merging independently specialized checkpoints in parameter space beats joint multi-source continued pretraining for domain adaptation, retaining complementary domain signals that joint training tends to wash out.
Natural Language Processing (NLP) in the climate domain requires models to process heterogeneous text sources, including scientific literature, policy disclosures, and synthetic reports. However, how to effectively combine diverse domain corpora during continued pretraining (CPT) remains underexplored. We introduce Climate-ModernBERT, a family of climate-adapted encoder models obtained through continued pretraining of ModernBERT-Base on three climate corpora: academic climate text, climate-filtered web data, and synthetic climate documents. We systematically compare joint continued pretraining on corpus mixtures with parameter-space merging of independently specialized checkpoints. Across nine climate NLP benchmarks, our best model achieves 76.3 average F_1, improving significantly over a vanilla ModernBERT baseline by 2.8 points. Within the climate NLP setting, the results show that academic climate corpora provide the strongest adaptation signal among the evaluated sources, while parameter-space merging improves over joint multi-source training and better preserves complementary information from heterogeneous climate corpora. We release all Climate-ModernBERT variants and training checkpoints to support future research in climate NLP and domain-adaptive pretraining.
