Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
The paper investigates domain fine-tuning of the Finnish BERT model (FinBERT) on Finnish medical text data, specifically histopathological reports, to improve performance in low-labeled-data scenarios. The authors analyze the geometric changes in embeddings during domain fine-tuning and explore their correlation with downstream task performance. They find that certain geometric properties of the embedding space after fine-tuning correlate with improved performance on downstream classification tasks, suggesting potential predictors for the effectiveness of domain adaptation.
Forget painstakingly labeling data – this study hints that you can predict the benefit of domain-specific BERT fine-tuning simply by observing how the embedding geometry shifts during training.
In NLP classification tasks where little labeled data exists, domain fine-tuning of transformer models on unlabeled data is an established approach. In this paper we have two aims. (1) We describe our observations from fine-tuning the Finnish BERT model on Finnish medical text data. (2) We report on our attempts to predict the benefit of domain-specific pre-training of Finnish BERT from observing the geometry of embedding changes due to domain fine-tuning. Our driving motivation is the common\situation in healthcare AI where we might experience long delays in acquiring datasets, especially with respect to labels.