Search papers, labs, and topics across Lattice.
This paper introduces a novel schema for evaluating errors made by LLMs in scholarly question-answering, designed to reflect the assessment strategies of domain experts. Through thematic analysis of question-answer pairs and contextual inquiries with scientists, the authors identified and validated 20 error patterns across seven categories, revealing expert evaluation strategies. The study demonstrates that structured evaluation schemas can help experts identify previously overlooked issues, paving the way for personalized, schema-driven evaluation tools.
LLM evaluation is missing the forest for the trees: automated metrics overlook critical errors that domain experts readily identify using nuanced, context-aware strategies.
Large Language Models (LLMs) are transforming scholarly tasks like search and summarization, but their reliability remains uncertain. Current evaluation metrics for testing LLM reliability are primarily automated approaches that prioritize efficiency and scalability, but lack contextual nuance and fail to reflect how scientific domain experts assess LLM outputs in practice. We developed and validated a schema for evaluating LLM errors in scholarly question-answering systems that reflects the assessment strategies of practicing scientists. In collaboration with domain experts, we identified 20 error patterns across seven categories through thematic analysis of 68 question-answer pairs. We validated this schema through contextual inquiries with 10 additional scientists, which showed not only which errors experts naturally identify but also how structured evaluation schemas can help them detect previously overlooked issues. Domain experts use systematic assessment strategies, including technical precision testing, value-based evaluation, and meta-evaluation of their own practices. We discuss implications for supporting expert evaluation of LLM outputs, including opportunities for personalized, schema-driven tools that adapt to individual evaluation patterns and expertise levels.
