Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This study investigates the effects of various pre-training objectives on the performance of encoder-decoder pre-trained language models in generation and question answering tasks, particularly focusing on commonsense knowledge retrieval. The authors introduce the Match Task to Objective (MTO) framework, which automates the selection of appropriate objectives for specific tasks and enhances both fine-tuning and prompt-tuning strategies. Results indicate that aligning these strategies with task requirements can yield over a 120% performance improvement in few-shot settings compared to traditional methods, significantly outperforming existing approaches even with full datasets.
Aligning pre-training objectives with specific tasks can boost performance by over 120% in few-shot scenarios, revolutionizing how we approach model fine-tuning.
Prompt-based learning has emerged as a dominant paradigm in natural language processing. This study explores the impact of diverse pre-training objectives on the performance of encoder-decoder pre-trained language models across generation and question answering tasks, with a focus on commonsense knowledge retrieval and completion. We highlight the benefits of incorporating multiple objectives during both pre-training and fine-tuning stages. We introduce the Match Task to Objective (MTO) framework and methods for determining the appropriate objective for a given task. This framework offers automated methods to prepare task-related data for adaptation through unsupervised training, based on the identified objective. In the fine-tuning stage, we design novel templates that align with the objectives of the pre-training and adaptation stages. When aligned with task requirements, these strategies can achieve a performance gain of over 120\% compared to conventional methods in few-shot settings. They significantly outperform related works in few-shot settings and exceed the baseline even in full-dataset scenarios. Furthermore, we extend this approach to include prompt-tuning methodologies, providing guidance for more effective soft prompt engineering and optimization. Our strategies significantly enhance prompt-tuning performance as well. These insights hold substantial value, precisely guiding the selection and optimization of models customized for specific tasks. Code is available at https://github.com/puraminy/MTO/