Search papers, labs, and topics across Lattice.
This paper introduces ACL-Verbatim, an extractive question answering system designed to eliminate hallucinations in AI-assisted research by mapping user queries to verbatim text spans in academic papers. By creating a novel ground truth dataset and employing a 150M-parameter ModernBERT token classifier, the authors demonstrate that their approach outperforms existing LLM extractors in accuracy, achieving a word-level F1 score of 53.6 compared to the best LLM's 48.7. This advancement is crucial for researchers who rely on precise information retrieval from trusted sources, ensuring the reliability of AI-generated content in academic contexts.
A 150M-parameter ModernBERT classifier outperforms leading LLMs in extractive question answering, achieving a significant accuracy boost for academic research.
Academic researchers need efficient and reliable methods for collecting high-quality information from trusted sources, but modern tools for AI-assisted research still suffer from the tendency of Large Language Models (LLMs) to produce factually inaccurate or nonsensical output, commonly referred to as hallucinations. We apply the extractive question answering system VerbatimRAG to research papers in the ACL Anthology, directly mapping user queries to verbatim text spans in retrieved documents. We contribute a novel ground truth dataset for the task of mapping user queries to relevant text spans in research papers, and use it to train and evaluate a variety of extractive models. Human annotation is performed by NLP researchers and is based on synthetic user queries generated using a custom pipeline based on the ScIRGen methodology, paired with chunks of research papers retrieved by VerbatimRAG. On this benchmark, a 150M-parameter ModernBERT token classifier trained on silver supervision from our pipeline achieves the best word-level F1 (53.6), ahead of the strongest evaluated LLM extractor (48.7).
