Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This study compared LLM-generated Japanese translations of chest CT reports with human-edited versions, evaluating them for educational suitability. Blinded radiologists and LLM judges (DeepSeek-V3.2, Mistral Large 3, and GPT-5) performed pairwise evaluations across terminology accuracy, readability, overall quality, and radiologist-style authenticity. Radiologists showed poor agreement with each other and favored either LLM or human translations depending on the criteria, while LLM judges overwhelmingly preferred LLM-generated translations, highlighting the limitations of using LLMs to evaluate LLM-generated content in specialized domains.
LLMs think their translations of medical reports are better than human translations, but radiologists disagree, revealing a critical flaw in using LLMs to self-evaluate specialized content.
Background: Accurate translation of radiology reports is important for multilingual research, clinical communication, and radiology education, but the validity of LLM-based evaluation remains unclear. Objective: To evaluate the educational suitability of LLM-generated Japanese translations of chest CT reports and compare radiologist assessments with LLM-as-a-judge evaluations. Methods: We analyzed 150 chest CT reports from the CT-RATE-JPN validation set. For each English report, a human-edited Japanese translation was compared with an LLM-generated translation by DeepSeek-V3.2. A board-certified radiologist and a radiology resident independently performed blinded pairwise evaluations across 4 criteria: terminology accuracy, readability, overall quality, and radiologist-style authenticity. In parallel, 3 LLM judges (DeepSeek-V3.2, Mistral Large 3, and GPT-5) evaluated the same pairs. Agreement was assessed using QWK and percentage agreement. Results: Agreement between radiologists and LLM judges was near zero (QWK=-0.04 to 0.15). Agreement between the 2 radiologists was also poor (QWK=0.01 to 0.06). Radiologist 1 rated terminology as equivalent in 59% of cases and favored the LLM translation for readability (51%) and overall quality (51%). Radiologist 2 rated readability as equivalent in 75% of cases and favored the human-edited translation for overall quality (40% vs 21%). All 3 LLM judges strongly favored the LLM translation across all criteria (70%-99%) and rated it as more radiologist-like in>93% of cases. Conclusions: LLM-generated translations were often judged natural and fluent, but the 2 radiologists differed substantially. LLM-as-a-judge showed strong preference for LLM output and negligible agreement with radiologists. For educational use of translated radiology reports, automated LLM-based evaluation alone is insufficient; expert radiologist review remains important.