Search papers, labs, and topics across Lattice.
This study investigates the limitations of existing scalar metrics in evaluating the clinical significance of radiology reports generated by Large Language Models (LLMs), highlighting a discrimination bias where models penalize harmless variations too harshly. Using the ReEvalMed benchmark, the authors synthesize 4,000 report pairs and train lightweight interpretable metrics on Qwen3-8B and MedGemma-4B, achieving superior performance in delineating clinically significant errors compared to larger models. The results indicate that a one-pass training approach is more effective and cost-efficient than a two-pass method, which compromises overall performance for robustness.
LLMs can misclassify harmless rephrasings as critical errors, but a new lightweight metric outperforms larger models in clinical significance evaluation.
Reliable evaluation of generated radiology reports requires strict clinical accuracy, as omitted critical findings or mischaracterized radiographic observations can directly affect patient care. Existing metrics obscure this requirement by reducing report quality to a medically ungrounded scalar. Although Large Language Models (LLMs) possess rich medical knowledge, they likewise struggle to draw a reliable boundary between clinically significant errors and harmless variation. We study this boundary using ReEvalMed benchmark as testbed and evaluate metric-level clinical significance from detecting true clinical errors ("Discrimination") and tolerating insignificant variations ("Robustness"). Across 8 LLM evaluators under one-pass and two-pass settings, we identify a widespread discrimination bias: models effectively detect errors but also over-penalize harmless rephrasings. To mitigate this, we synthesize 4k report pairs and train lightweight interpretable metrics on Qwen3-8B and MedGemma-4B. Our trained metric sharpens the clinical significance boundary, surpassing 32B-scale medical LLMs and remaining competitive with proprietary models. Crucially, the more costly two-pass setting fails to consistently improve overall performance and mainly trades discrimination for robustness. These findings suggest one-pass trained metrics as the practical choice for cost-sensitive deployment, with two-pass inference reserved for settings where D-R balance is critical. We will release the dataset and metric.
