Search papers, labs, and topics across Lattice.
This study evaluates the ability of large language models (LLMs) to identify and correct misconceptions in multi-turn medical conversations using a newly introduced dataset, ThReadMed-QA, which consists of 2,437 patient-physician dialogue threads. The findings reveal that while models like GPT-5 and Claude-Haiku can accurately correct misconceptions approximately 85% of the time in initial interactions, their performance drops to around 50% after just two follow-up questions due to error propagation. This degradation underscores the importance of developing evaluation frameworks that account for multi-turn interactions, as inconsistent model performance could lead to unsafe medical guidance for patients.
Even top-performing LLMs struggle to maintain accuracy in correcting medical misconceptions, with performance plummeting from 85% to 50% over just two follow-up questions.
Patients seeking medical information often ask questions that embed incorrect assumptions or misconceptions. In such cases, safe medical communication requires not only answering the question, but identifying and correcting the underlying false belief. These interactions naturally unfold over multiple turns, a pattern now mirrored in interactions with LLMs. Yet current evaluation frameworks do not capture model behavior in these settings, where misconceptions can emerge, persist, or evolve over the course of a conversation. Whether LLMs can reliably correct such misconceptions over time remains largely unexamined. To study this, we introduce ThReadMed-QA, a multi-turn medical dialogue dataset of 2,437 patient-physician conversation threads comprising 8,204 question-answer pairs, derived from real patient interactions on AskDocs. This dataset enables systematic evaluation of whether models can detect and correct misconceptions under a multi-turn context. We evaluate five LLMs using a rubric-based LLM-as-a-Judge framework that scores responses based on their ability to identify and correct misconceptions. Our experiments reveal a consistent pattern: even frontier models that can address misconceptions in a single interaction degrade substantially over subsequent turns. GPT-5 and Claude-Haiku correct these false presuppositions around 85% on initial questions but drop to roughly 50% within two follow-ups. An oracle analysis replacing prior model outputs with physician responses shows that much of the degradation is driven by error propagation, while performance remains imperfect even under correct context. Even when models tend to correct misconceptions initially, their performance degrades substantially over later turns, leading to inconsistent and potentially unsafe guidance in patient-facing settings and highlighting the need for evaluation frameworks that capture multi-turn behavior.
