Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
The authors introduce a pipeline leveraging LLMs and terminology validation to generate clinically realistic HL7 FHIR R4 bundles from unstructured text, creating a new dataset called MedCase-Structured. This dataset is derived from MedCaseReasoning and is designed to benchmark clinical decision support systems using structured, interoperable data formats. Experiments show that LLMs exhibit lower diagnostic accuracy when processing structured FHIR inputs compared to plain text, underscoring the need for benchmarks that reflect real-world clinical data formats.
LLMs that ace medical reasoning on text bomb on structured EHR data, revealing a critical gap in deployment readiness.
Large language models (LLMs) show promise for clinical reasoning and decision support, but evaluation in realistic, electronic health record-congruent settings remains limited. Existing benchmarks often rely on static datasets or unstructured inputs that do not reflect the structured, interoperable data formats used in clinical systems. We introduce a pipeline for generating clinically realistic HL7 FHIR R4 bundles from unstructured text, enabling controllable evaluation of clinical decision support systems. The pipeline combines staged LLM generation with terminology-grounded validation and repair to reduce hallucinated codes and enforce structural and semantic consistency. Applying this approach to MedCaseReasoning, we construct MedCase-Structured, a synthetic dataset aligned with clinician-authored diagnostic cases, achieving valid FHIR generation for 82.5% of cases. Evaluation on MedCase-Structured reveals consistently lower diagnostic accuracy for LLMs on structured FHIR inputs than with plain text, highlighting the importance of deployment-aligned benchmarking.