Search papers, labs, and topics across Lattice.
This study introduces MedQADE, a standardized open-response clinical benchmark for German, featuring 3,800 items evaluated by both practicing physicians and Large Language Model (LLM) evaluators. The top LLM evaluator, Gemini 3 Flash, achieved a kappa score of 0.694, closely aligning with the physician ceiling of 0.709, yet exhibited a concerning lack of clinical metacognition, scoring definitively on all items regardless of difficulty. Additionally, the research uncovered systematic biases where models favored scoring their architectural siblings, highlighting that statistical alignment does not equate to clinical caution and emphasizing the need for independent verification of evaluator performance.
Statistical alignment with clinical experts doesn't guarantee that LLMs exercise the necessary clinical caution in evaluations.
Open-response evaluation provides stronger clinical validity than multiple-choice benchmarks but creates a scoring bottleneck that motivates automated LLM-asa-Judge approaches. Whether such evaluators replicate clinical calibration and caution, however, remains untested. We introduce MedQADE, the first standardised open-response clinical benchmark for German, a major clinical language lacking native evaluation infrastructure, comprising 3,800 items annotated by ten practising physicians and nine Large Language Model (LLM) evaluators. The top-performing evaluator model, Gemini 3 Flash, reached alignment consistent with the physician ceiling (\k{appa} = 0.694 vs. \k{appa} = 0.709), though wide confidence intervals limit interpretation. Despite this statistical alignment, automated evaluators exhibited near-absent clinical metacognition: physicians scaled abstention with item difficulty, while frontier models assigned definitive scores in every case. We additionally quantified systematic lineage-dependent biases, where models preferentially scored architectural siblings, an effect independent of language. These results show that statistical alignment does not ensure clinical caution, and that evaluator independence requires explicit verification.
