Search papers, labs, and topics across Lattice.
This paper introduces the Maskability Index (MI), a quantitative metric designed to assess the alignment between prompting strategies and pretraining objectives in large-scale pretrained language models like T5 and BERT. By analyzing DepthRank score differences for masked versus unmasked templates, the authors demonstrate that MI effectively predicts downstream generation performance across various knowledge relations from the ATOMIC2020 benchmark. The findings reveal that leveraging MI can enhance the selection of prompting templates and adaptation strategies, particularly in low-resource scenarios, thereby improving the extraction of relational knowledge from these models.
The Maskability Index reveals that the right prompting strategy can significantly boost the performance of pretrained language models in knowledge extraction tasks.
Large-scale pretrained language models such as T5 and BERT have demonstrated strong capabilities for generating structured knowledge. However, their performance depends on how closely the prompting strategy matches the objectives used during pretraining. We introduce the Maskability Index (MI), a quantitative metric that estimates whether a knowledge relation is better suited to masked-style prompting or prefix-style prompting in few-shot generation. MI is computed from differences in DepthRank scores between masked and unmasked templates, providing a principled measure of objective-template alignment. We evaluate MI on a diverse set of relations from the ATOMIC2020 knowledge base completion benchmark and show that it is positively correlated with downstream generation performance. These results indicate that MI can help select appropriate prompting templates and adaptation strategies for extracting relational knowledge from pretrained language models, especially in low-resource settings.
