Search papers, labs, and topics across Lattice.
This study evaluates three LLM-based methods鈥擥PT-4o, Mistral, and a bespoke algorithm called DSIT-Taxonomies鈥攆or extracting and classifying research entities from UKRI grant proposals. The research aims to identify early signals of emerging research areas to guide public investment, revealing that Mistral and GPT-4o yield comparable entity sets with significant semantic overlap, while Mistral achieves a notable topic classification accuracy of 90.5%, significantly outperforming DSIT-Taxonomies at 71.4%. These findings suggest that Mistral is a superior choice for large-scale analysis of sensitive grant data, offering both operational efficiency and security.
Mistral outperforms existing methods with a striking 90.5% accuracy in topic classification, revealing a new standard for entity extraction in grant proposals.
This paper presents preliminary findings from a UKRI-funded Metascience project comparing three LLM-based approaches, GPT-4o, Mistral, and a bespoke algorithm, DSIT-Taxonomies, for extracting and classifying research entities from funding proposals. Our project"Tracking Stars and Unicorns"aims to identify early signals of emerging research areas to inform public investment. Our methodology employed a three-stage pipeline, leveraging Mistral for primary entity extraction and mapping against the OpenAlex Topics taxonomy. We evaluated our approach across 42 proposals'abstracts from different areas and observed that Mistral and GPT-4o produce comparable, high-quality entity sets with significant semantic overlap, outperforming the fragmented DSIT-Taxonomies approach. Crucially, the Mistral-based approach achieved superior topic classification accuracy (90.5%) compared to the full DSIT-Taxonomies pipeline (71.4%). We conclude that Mistral offers a high-performance, operationally efficient, and secure solution for large-scale analysis of sensitive grant data.
