Search papers, labs, and topics across Lattice.
This paper introduces a vision-language model pre-trained on CT images and radiology reports using visual instruction tuning to improve survival prediction. The model learns clinically relevant visual-textual representations from 3D CT scans, which are then fine-tuned for survival prediction using a dedicated prediction head. Experiments show that the proposed method outperforms baselines, especially when clinical data is limited, suggesting the model effectively leverages visual information from CT scans.
Freeing survival prediction from reliance on expert-derived features, this model learns directly from CT scans and radiology reports to improve accuracy, especially when clinical data is scarce.
Accurate prognostication and risk estimation are essential for guiding clinical decision-making and optimizing patient management. While radiologist-assessed features from CT scans provide valuable indicators of disease severity and outcomes, interpreting such images requires expert knowledge, and translating rich visual information into textual summaries inevitably leads to information loss. In this work, we propose a vision-language framework for 3D CT image understanding that leverages large-scale open-sourced CT images paired with radiology reports through visual instruction tuning. This pre-training enables the model to learn clinically meaningful visual-textual representations, which can then be adapted to downstream survival prediction tasks. By incorporating a survival prediction head on top of the pre-trained model, our approach improves survival prediction from CT images and clinical data while generating clinically meaningful language responses to predefined questions. Experimental results demonstrate that our method outperforms baseline methods in survival prediction, particularly, when clinical data alone is less predictive. The code will be released upon acceptance.
