Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces MM-KG, a novel multimodal knowledge graph that explicitly links heterogeneous patient observations (EHR text, imaging, genomics, biospecimens) to biomedical concepts via typed alignment edges. By representing these as separate layers within a unified graph and employing query-conditioned retrieval, MM-KG enables traceable and testable reasoning for clinical LLMs. The approach significantly improves performance on questions requiring both patient evidence and external knowledge, demonstrating a substantial AUROC interaction gain and outperforming existing methods on held-out ranking tasks.
Explicitly linking multimodal patient data to biomedical knowledge graphs unlocks significant gains for clinical LLMs, with performance plummeting when these crucial connections are removed.
Clinical questions often depend on linking a patient's multimodal evidence to external biomedical knowledge, yet existing predictive systems rarely represent such links explicitly, so they can neither be traced to their evidence sources nor removed to measure their contributions. We present MM-KG (Multimodal Knowledge Graph), which represents heterogeneous, multimodal patient observations and biomedical concepts as separate layers in one typed graph, joined by explicit alignment edges. First, modality-specific harmonizers convert EHR text, imaging, genomic, and biospecimen data into typed observations mapped to UMLS concepts, which a route-prioritized aligner links to a biomedical knowledge graph. Query-conditioned retrieval then selects a compact subgraph for downstream use by a large language model or a graph neural network. We build MM-KGs for MIMIC-IV and ADNI, and evaluate them with a 2x2 design that separates patient evidence, biomedical knowledge, and their interaction. On questions that require both sources, neither source alone performs far above chance, whereas their combination yields a drug-controlled AUROC interaction of +0.194 on MIMIC and +0.299 on ADNI. On held-out five-candidate ranking, MM-KG outperforms MindMap by +0.131 Hits@1 and leads an adapted GraphCare on the items that require consulting the patient, and deleting the single answer-bearing relation from the retrieved packet returns Hits@1 to the no-knowledge baseline. Finally, query-conditioned retrieval reaches 0.731 AUROC with 6.8x less context than the strongest generic policy, whereas static knowledge graph context gives no consistent gain on ordinary outcome prediction. Knowledge graphs thus benefit clinical LLMs not as background context but as explicit links between multimodal patient evidence and the relation a question requires, and MM-KG makes these links retrievable, traceable, and testable.