Search papers, labs, and topics across Lattice.
This study introduces the TRACES benchmark, which evaluates the epistemic reliability of large language models (LLMs) in distinguishing between reliable and unreliable scientific literature, a critical capability for their deployment in scientific workflows. By utilizing a probe corpus of 42 problematic papers and scoring model engagement through two metrics鈥擨FR-a and IFR-i鈥攔esearchers found that LLMs overwhelmingly engage with flawed premises, with over 95% of responses failing to reject untenable claims. The findings reveal that all evaluated models exhibit significant deficiencies in epistemic competence, failing more than 71% of the time on agentic probes, highlighting an urgent need for safety measures in LLM applications in science.
LLMs engage with flawed scientific premises 95% of the time, revealing a critical gap in their epistemic reliability for scientific applications.
Large language models are being proposed as agents in scientific workflows, in domains where no downstream verifier exists. Such deployment assumes the model can distinguish reliable scientific literature from unreliable literature, a capability that has not yet been directly measured. Existing benchmarks evaluate factuality on questions with known answers; the failure mode we target here is different. We introduce a probe corpus of 42 retracted, fraudulent, and pseudoscientific papers, paired with a methodology for eliciting and scoring single-shot model engagement with each paper's framing. Each probe pairs a preamble extracted near-verbatim from the target paper with a scientifically plausible study-design request. The probes span five claim types: fabricated observation, pseudophysical mechanism, magical premise, legitimization bridge, and cargo-cult experiment. Two complementary scores measure whether a model rejects the flawed premise outright (IFR-a) and whether it recognizes the unreliability while still engaging (IFR-i). A depth score, the Engagement Depth Index (EDI), quantifies reproduction of paper- or field-specific withheld details. Across 30 models and 10 repeated runs, aggregate IFR-a is 0.93 $\pm$ 0.004 and aggregate IFR-i is 0.809 $\pm$ 0.009. Models engaged with untenable premises in 95% of all non-empty responses. Every evaluated model fails more than 71% of agentic probes, and 22 of 30 models fail more than 90% of the time. Rejections are concentrated on a small number of high-notoriety topics and specific probes, and disappear under matched-structure controls. These results are consistent with topic-keyed safety behavior rather than robust epistemic competence, and indicate an urgent need for guardrail infrastructure for scientific deployment of language models.
