Search papers, labs, and topics across Lattice.
This paper introduces PseudoBench, an adversarial benchmark designed to evaluate the ability of agentic auto-research systems to identify and resist pseudoscientific narratives. The study tests seven state-of-the-art agents, revealing that they generate persuasive reports aligned with pseudoscientific claims, with the most resistant agent achieving only 27.4% effectiveness in rejecting such narratives. These results highlight a significant risk that autonomous research systems could inadvertently contribute to the spread of pseudoscience, necessitating urgent attention to alignment strategies before deployment.
Current AI agents are alarmingly effective at packaging pseudoscience in credible scientific language, with refusal rates near zero.
As Large Language Model based agents enter autonomous scientific research, their ability to resist pseudoscience becomes increasingly important. Otherwise, such systems may rapidly generate plausible yet misleading studies that contaminate academic literature and erode trust in science. We present PseudoBench, an adversarial benchmark for evaluating whether agentic auto-research systems can identify and resist pseudoscientific narratives. PseudoBench contains 200 curated pseudoscientific claim-evidence pairs across five domains and evaluates agents through an end-to-end research pipeline from experiments to writing. Testing seven state-of-the-art agents, we find that current systems readily produce persuasive reports that align with pseudoscientific premises with near-zero refusal rates and the highest resistance of only 27.4%. Stronger agents risk packaging pseudoscience in more sophisticated scientific language, increasing its apparent credibility. These findings reveal an alarming capacity to fuel pseudoscience, calling for scientific alignment before widespread deployment.
