Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces an attention expansion mechanism that enhances keyphrase extraction (KPE) from long documents by augmenting pre-trained language model (PLM) token representations with contextual information from out-of-context chunks using pre-trained word embeddings. This approach addresses the limitations of traditional PLMs, which struggle with distant salient keyphrase evidence due to their restricted context windows, while avoiding the high computational costs associated with long-context large language models. Experimental evaluations across various PLM backbones and benchmark corpora demonstrate that attention expansion significantly improves KPE performance, achieving superior F1 scores compared to state-of-the-art models.
Attention expansion boosts keyphrase extraction performance by leveraging out-of-context information, outperforming existing models without the computational burden of full-document attention.
Pre-trained language models (PLMs) have achieved strong performance in keyphrase extraction (KPE), largely due to their ability to generate rich contextualized representations. However, long-document KPE remains challenging because salient keyphrase evidence may be scattered across distant document sections that cannot be jointly captured within the limited context window of most PLMs. Although long-context large language models (LLMs) can process broader textual contexts, their computational cost limits their practicality for efficient and high-throughput KPE. To overcome this limitation, we propose an attention expansion mechanism that augments PLM token representations with information from surrounding out-of-context chunks using pre-trained word embeddings. The proposed mechanism expands the effective contextual scope of PLM-based KPE models without requiring full-document attention or expensive LLM-based inference. We evaluate our approach across five PLM backbones, including general-purpose, scientific, task-specific, and long-context encoders, using two training regimes and five benchmark corpora from scientific and news domains. Experimental results demonstrate that attention expansion consistently enhances KPE performance across all evaluation settings, outperforming state-of-the-art models and yielding notable improvements in F1 score. The improvements extend to domain-specific, task-specialized, and native long-context models, showing that the proposed mechanism provides complementary information rather than merely compensating for limited input length. These results establish attention expansion as an efficient and effective strategy for long-document KPE.