Search papers, labs, and topics across Lattice.
This paper introduces MedAgent-R1, a reinforcement learning-based medical reasoning agent designed to enhance the faithfulness of clinical justifications while maintaining accuracy. The authors identify a critical issue termed "confident hallucination," where traditional outcome-only rewards lead to increased citation fabrication rates, undermining the reliability of AI-generated medical advice. By implementing a faithfulness-gated reward system, MedAgent-R1 significantly reduces citation fabrication from 31.8% to 4.7% and improves evidence completeness, demonstrating that explicit faithfulness training can yield substantial benefits over mere model scaling.
Faithfulness-aware training can slash citation fabrication rates from nearly one-third to just 4.7%, transforming the reliability of AI in clinical settings.
When medical AI systems hallucinate clinical reasoning, the consequences extend beyond incorrect answers: fabricated justifications that superficially reference retrieved evidence can mislead clinicians into unsafe treatment decisions. Medical reasoning agents must therefore produce not only correct answers but also faithful justifications that clinicians can verify against cited evidence. We identify a systematic failure mode in RL-trained retrieval agents: outcome-only rewards improve accuracy while degrading faithfulness, a phenomenon we term confident hallucination. The agent learns to answer from parametric memory and backfill plausible but unsupported justifications; citation fabrication rates rise from 16.5% to 31.8% even as accuracy improves by 5 points over the supervised baseline. We address this with a faithfulness-gated reward design: accuracy credit is conditioned on evidence grounding via a hard gate, complemented by retrieval validity and conciseness signals that close exploitation paths unique to agentic retrieval. The resulting system, MedAgent-R1, reduces citation fabrication from 31.8% to 4.7% and raises evidence completeness from 58.7 to 82.6 while maintaining 75.1% accuracy, with 13.2-point gains on HealthBench Safety. Under the same agentic retrieval setup, MedAgent-R1 outscores GPT-4o on faithfulness-specific dimensions (Factual Support 4.55 vs. 4.25; Overclaiming 4.40 vs. 4.15) while remaining below GPT-4o in overall accuracy, suggesting that explicit faithfulness training yields evidence-grounding gains not achieved by scaling alone.
