Search papers, labs, and topics across Lattice.
This study conducts a comprehensive evaluation of five state-of-the-art large language models (LLMs) for citation function classification, achieving new benchmarks on the ACL-ARC dataset. The fine-tuned Falcon 7B model reached a macro F1 score of 73.3%, significantly outperforming previous methods. Additionally, the introduction of the AC3 dataset, which features a nuanced seven-category annotation scheme, enables a deeper understanding of citation functions, particularly in distinguishing between neutral acknowledgments and evaluative stances.
Fine-tuning the Falcon 7B model not only sets a new benchmark in citation function classification but also reveals the critical role of contextual scope in model performance.
Citation function classification plays a crucial role in understanding the relationships between scientific publications and advancing bibliometric analysis. This study presents one of the first comprehensive evaluations of multiple state-of-the-art (SOTA) large language models (LLMs) for citation function classification, achieving new SOTA results on the ACL-ARC dataset. We systematically compare five models (Mistral 7B, Orca 2-7B, LLaMA 3.1-8B, Falcon 7B, and SciBERT) across zero-shot, few-shot, and fine-tuning approaches. Our fine-tuned Falcon 7B model achieves a 73.3% macro F1 score on ACL-ARC, representing a significant improvement over previous methods. Additionally, we introduce AC3, a novel dataset featuring a seven-category annotation scheme that distinguishes between neutral acknowledgments and explicit evaluative stances (more opinion-oriented citations - criticizing, complimenting, contradicting). The dataset is implemented across four context extraction variants to systematically evaluate the impact of contextual scope on classification performance. We also provide detailed analysis of model performance, experimental configurations, and limitations to guide future research in this domain. To our knowledge, this is one of the first studies dedicated to comprehensive model comparison for citation function classification, addressing a gap identified in recent surveys.
