Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces ResearchQA, a benchmark designed to evaluate citation-grounded question-answering capabilities of large language models using a dataset of 6,211 question-answer pairs derived from 494 scientific papers across various domains. The benchmark emphasizes the importance of verifiable citations by rewarding models that provide grounded refusals when answers lack support from the source material. Key findings reveal that citation-based metrics offer clearer differentiation between model performance compared to traditional LLM evaluator scores, with open-weight models achieving competitive citation accuracy while significantly reducing latency.
Citation-grounded evaluation reveals that open-weight models can rival closed models in accuracy while being three to six times faster.
Large language models are increasingly used to assist scientific reading, but existing evaluation methods often fail to detect whether answers are supported by verifiable citations. We introduce ResearchQA, a benchmark of 6,211 single-paper question-answer pairs from 494 open-access papers spanning eight domains and four question types: lookup, comprehension, multi-hop, and adversarial. ResearchQA is designed for citation-grounded evaluation: it permits multiple valid supporting passages for a claim and rewards grounded refusal when the source paper does not support an answer. We evaluate eight leading closed- and open-weight models in a citation-grounded chat-with-paper setting using a deterministic citation matcher and an LLM-based rubric evaluator. Citation-based metrics separate systems more clearly than LLM-evaluator scores: section coverage and citation accuracy vary substantially across models, while evaluator scores remain tightly compressed. We further find that open-weight models approach the best closed-model citation accuracy while achieving 3 to 6 times lower per-example latency. We release the benchmark, evaluation harness, and evaluator prompt.