Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This study evaluates the effectiveness of various evidence-generating language models (LLMs) for biomedical claim verification using the CARE-XAI benchmark, which encompasses multiple health fact-checking sources. The findings reveal that while biomedical classifiers excel at verdict-only predictions, fine-tuned LLMs outperform others in generating useful evidence; however, the utility of PubMed retrieval is context-dependent, enhancing performance for specific sources but potentially distracting for broader claims. The introduction of Bio-GRACE offers a novel diagnostic tool to assess the recovery of decision benefits from retrieved evidence, highlighting the limitations of traditional metrics in biomedical fact-checking.
Fine-tuned LLMs excel in generating evidence for biomedical claims, but PubMed retrieval can mislead on broader public health topics.
Biomedical fact-checking systems must do more than predict whether a claim is supported, contradicted, or unaddressed: they should also produce evidence that is faithful, complete, and useful for verification. We study this evidence-generation setting on CARE-XAI, a unified benchmark spanning five biomedical and health fact-checking sources. We compare base instruction LLMs, PubMed retrieval-augmented LLMs, fine-tuned LLMs, label-only LLMs, and biomedical encoder classifiers under a shared evaluation protocol. Biomedical classifiers remain strongest for verdict-only prediction, while fine-tuned LLMs are the strongest evidence-generating systems. PubMed retrieval is mixed: it helps PubMed-aligned sources such as PubMedQA and SciFact, but can distract models on broader public-health claims. We introduce Bio-GRACE, a gold-reference-normalized diagnostic for measuring whether retrieved evidence recovers the decision benefit of reference evidence. Bio-GRACE shows that retrieval utility is source-dependent, motivates selective retrieval, and exposes why retrieval recall and lexical evidence overlap are insufficient for biomedical fact-checking.