Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This study introduces KliniskVestBERT, a suite of three BERT-based models specifically pre-trained on a large corpus of Norwegian clinical texts sourced from Helse Vest, including diverse document types such as discharge summaries and nursing notes. By continuing the pretraining of existing models—Nb-BERT-large, NorBERT3-large, and ModernBERT—on this specialized dataset, the authors demonstrate significant performance improvements on both synthetic and real-world clinical NLP tasks. The results underscore the importance of domain-specific pre-training, as KliniskVestBERT consistently outperforms baseline models, highlighting its potential for enhancing NLP applications in healthcare settings.
KliniskVestBERT outperforms existing models in clinical NLP tasks, showcasing the power of domain-specific pre-training in healthcare applications.
The increasing application of Natural Language Processing (NLP) in healthcare demands language models specifically attuned to the complexities of clinical language. This work introduces KliniskVestBERT, a suite of three BERT-based encoder models pre-trained on a substantial corpus of real-world, de-identified Norwegian clinical texts from Helse Vest. We continue pretraining existing language models Nb-BERT-large, NorBERT3-large, and ModernBERT on our specialized clinical dataset. This dataset is based on a representative population of Helse Vest patients. The included document types are carefully curated to encompass a broad clinical spectrum in bokmål and nynorsk including discharge summaries, surgical reports, nursing notes etc. ensuring comprehensive representation of the linguistic landscape within Norwegian healthcare settings. Evaluation on three synthtetic Norwegian clinical benchmark datasets and two real-world problems demonstrates that each of our clinically specialized models consistently outperforms their baseline counterparts, highlighting the significant benefit of domain-specific pre-training for NLP tasks within the clinical domain. The project was a joint effort by all Helse Vest entities (Helse Bergen, Helse Fonna, Helse Førde and Helse Stavanger) with DIPS under the project lead of Helse Vest ICT.