Search papers, labs, and topics across Lattice.
This study explores the adaptation of large language models (LLMs) to Swedish journalism through continued pre-training on a curated dataset of millions of news articles. The authors find that while continued pre-training enhances generation quality and factual knowledge, it requires experience replay to prevent forgetting, and shows limited impact on discriminative tasks. Additionally, the research highlights the necessity of targeted evaluation, revealing that existing benchmarks may not adequately reflect in-domain performance improvements.
Continued pre-training on Swedish journalism data significantly boosts LLMs' generation quality, but only when combined with experience replay to combat forgetting.
Large language models are increasingly capable in general, but their utility can remain modest in niche or understudied areas. One approach to address this limitation is to specialise existing models through additional training on target-domain corpora. In this work, we investigate such continued pre-training for adapting large language models to Swedish journalism, using a high-quality dataset that we curate from millions of news articles. To evaluate the adaptation efficacy, we also construct a novel domain-specific benchmark that covers six editorial tasks. Through full and parameter-efficient fine-tuning across two model sizes, we find that continued pre-training yields benefits in the target domain, but only when paired with experience replay to mitigate forgetting. We observe consistent enhancements in the models' generation quality and factual knowledge, but not their proficiency in discriminative tasks. Exploring a training-free method to facilitate instruction following, we see further improvements, but exclusively for models trained with low-rank adaptation. Crucially, we demonstrate the importance of targeted evaluation in the adaptation process, as an existing Swedish benchmark largely fails to capture the models' in-domain performance gains.
