Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This study investigates task-adaptive continual pre-training (TAPT) to enhance the classification capabilities of transformer-based language models specifically for Turkish language understanding. By focusing on the masked-language-modeling rate during continual pre-training on task-specific unlabeled text, the researchers achieved an average performance increase of 2.7% in classification tasks. These findings highlight the effectiveness of TAPT in improving domain-specific knowledge without the need for external domain corpora.
Task-adaptive continual pre-training can boost classification performance by 2.7% without relying on external domain data.
Transformer-based language models have become the standard in Natural Language Processing (NLP). They have surpassed human performance on specific classification tasks such as named-entity recognition, question-answer, text categorization, or generative tasks such as machine translation and summarization. However, since language models are trained with significant general-purpose texts, they may have limitations in their domain-specific knowledge. Techniques such as domain adaptation can be used to improve the models to address this issue. This study presents a systematic empirical investigation of task-adaptive continual pre-training (TAPT), introduced by Gururangan et al., for Turkish language understanding, with a particular focus on the effect of the masked-language-modeling rate. Adaptation is performed in the task-adaptive setting (TAPT), i.e., continual pre-training on the unlabeled text of the target task corpus, without requiring an external domain corpus. We achieved successful results with an average increase of 2.7%. We also addressed various issues and findings related to adaptation.