Search papers, labs, and topics across Lattice.
This study evaluates the effectiveness of large language models (LLMs) in conducting literature reviews for scientific research in physics, astrophysics, and cosmology through a controlled comparison of AI-generated and human-selected references across eight expert-defined projects. The analysis reveals that the overlap between human and AI-selected literature is minimal, at less than 6%, highlighting the current limitations of LLMs in independently performing competent literature searches. However, the findings also indicate that while mid-2025 models exhibit a notable rate of inaccuracies, the subsequent 2026 model demonstrates significant improvements, achieving zero errors in a single-project test.
AI-generated literature reviews miss the mark, with less than 6% overlap with expert selections, but the 2026 model shows promise for error-free performance.
We investigate how well large language models (LLMs) can assist with literature reviews for scientific research. We perform a controlled study of eight expert-conceived research projects across the areas of physics, astrophysics, and cosmology. Each project has a defined background and goal, and human experts and AI prompters are asked to perform identical literature review tasks in parallel. We compare the relevant literature selected by humans with that selected by mid-2025 LLMs (ChatGPT-4o, ChatGPT Deep Research, and Gemini). We find the overlap between human- and AI-selected references to be small ($<$6\%), indicating that AI models do not yet reproduce a competent expert search on their own, though they have the potential to complement literature searches by humans. We then assess the reliability and completeness of AI-generated candidate references, distinguishing two types of hallucination: fabrications (references to nonexistent papers) and metadata mismatches (real papers with one or more incorrect fields). We find that while fabricated references make up 3\% of the AI-generated references, 64\% are real papers with at least one incorrect field (title, author, year, journal, DOI, or link), indicating that the mid-2025 models require systematic verification. However, the performance is significantly improved for the 2026 model ChatGPT Pro 5.5, with a single-project test showing zero fabrication or metadata mismatches.
