Search papers, labs, and topics across Lattice.
This paper explores using LLMs to generate and evaluate synthetic French OSCE dialogues, addressing the scarcity of annotated French medical interview transcripts. A pipeline was developed to produce synthetic doctor-patient interview transcripts, simulating varying student skill levels based on scenario-specific evaluation criteria. Benchmarking results show that mid-size LLMs achieve accuracies comparable to GPT-4o on the synthetic data, suggesting the potential for locally deployable evaluation systems.
Mid-size LLMs can achieve near-GPT-4o accuracy in evaluating synthetic medical interviews, opening the door to privacy-preserving, locally deployable clinical skills assessment.
Objective Structured Clinical Examinations (OSCEs) are the standard method for assessing medical students'clinical and communication skills through structured patient interviews. In France, however, the organization of training sessions is limited by human and logistical constraints, restricting students'access to repeated practice and structured feedback. Recent advances in Natural Language Processing (NLP) and Large Language Models (LLMs) now offer the opportunity to automatically evaluate such medical interviews, thereby alleviating the need for human examiners during training. Yet, real French OSCE annotated transcripts remain extremely scarce, limiting reproducible research and reliable benchmarking. To address these challenges, we investigate the use of LLMs for both generating and evaluating French OSCE dialogues in a low-resource context. We introduce a controlled pipeline that produces synthetic doctor-patient interview transcripts guided by scenario-specific evaluation criteria, combining ideal and perturbed performances to simulate varying student skill levels. The resulting dialogues are automatically silver-labeled through an LLM-assisted framework supporting adjustable evaluation strictness. Benchmarking multiple open-source and proprietary LLMs shows that mid-size models ($\le$32B parameters) achieve accuracies comparable to GPT-4o ($\sim$90\%) on synthetic data, highlighting the feasibility of locally deployable, privacy-preserving evaluation systems for medical education.
