Search papers, labs, and topics across Lattice.
This study constructs a multimodal dataset of 1,000 academic papers that includes text, images, audio, and corresponding keywords to address the limitations of traditional keyword extraction methods that rely solely on textual data. By employing both unsupervised and supervised keyword extraction techniques, the research reveals that different modalities contribute uniquely to the extraction process, enhancing the model's ability to learn richer representations. The key finding demonstrates that integrating text from images and audio with traditional paper text significantly improves keyword extraction performance.
Combining text, images, and audio from academic papers can dramatically boost keyword extraction accuracy, revealing the untapped potential of multimodal data.
Up to this point, keyword extraction task typically relies solely on textual data. Neglecting visual details and audio features from image and audio modalities leads to deficiencies in information richness and overlooks potential correlations, thereby constraining the model's ability to learn representations of the data and the accuracy of model predictions. Furthermore, the currently available multimodal datasets for keyword extraction task are particularly scarce, further hindering the progress of research on multimodal keyword extraction task. Therefore, this study constructs a multimodal dataset of academic paper consisting of 1000 samples, with each sample containing paper text, images, audios and keywords. Based on unsupervised and supervised methods of keyword extraction, experiments are conducted using textual data from papers, as well as text extracted from images and audio. The aim is to investigate the differences in performance in keyword extraction task with respect to different modal information and the fusion of multimodal information. The experimental results indicate that text from different modalities exhibits distinct characteristics in the model. The concatenation of paper text, image text and audio text can effectively enhance the keyword extraction performance of academic papers.
