Search papers, labs, and topics across Lattice.
This paper introduces DecSelfMask, a novel approach that enhances decoder-only models for classification tasks by utilizing a self-relevance-guided masking strategy on unlabeled text. By leveraging relevance attribution methods, the model identifies and masks out non-relevant portions of unannotated texts to create self-supervised training examples, which are then used to improve downstream performance. The results demonstrate significant improvements across 136 tasks, achieving up to a 19.9-point increase in Macro F1 compared to traditional supervised fine-tuning methods.
Unlabeled data can be transformed into powerful training examples, boosting classification performance by nearly 20 points without the need for costly annotations.
Classification tasks require annotated data, which can often be expensive, time-consuming, or even unfeasible to collect. This is the case of the medical domain, where large datasets often have few annotated examples. To address this, we propose DecSelfMask (Decoder Self-learning by Masking), an approach to enhance decoder-only performance on classification tasks. We build on common self-learning approaches by leveraging a model to create training examples from unlabeled data to propose a novel relevance-guided masking strategy. We use relevance attribution methods to determine what portions of unannotated texts are relevant for a task. We then create self-supervised training examples by masking out those portions, training the model to reconstruct them via next-token-prediction. We hypothesize that those examples convey knowledge about the structure and semantics of unannotated data that can be useful for downstream performance. We test our approach on 136 tasks from a collection of 1.9M clinical notes from an Italian hospital. We quantify DecSelfMask's impact on downstream tasks on 5 models of different scales and families, including a probing analysis. Experiments show consistent gains, outperforming standard supervised fine-tuning approaches (+19.9 points in Macro F1), synthetic label generation (+12.5), and continual pretraining (+6.3), as well as common baselines.
