Search papers, labs, and topics across Lattice.
This paper introduces ClinPRISM, a novel multimodal LLM framework designed for effective question answering over irregular clinical time series (ICTS). By employing an irregularity-aware multi-scale encoder and a temporal evidence distiller, ClinPRISM captures and compresses sparse clinical data into a manageable format for LLMs, while a progressive alignment strategy ensures compatibility with textual embeddings. The framework demonstrates state-of-the-art performance with only 16 time-series tokens and an average inference latency of 0.15 seconds per question, highlighting its efficiency and effectiveness in clinical applications.
ClinPRISM achieves state-of-the-art QA performance on irregular clinical time series using just 16 tokens, revolutionizing efficiency in healthcare AI.
Question answering (QA) over irregular clinical time series (ICTS) plays a pivotal role in a wide range of healthcare applications. Although recent multimodal time-series large language models (LLMs) have shown considerable promise in general-purpose time-series QA, they remain poorly equipped to model the sparsity, asynchrony, and irregular sampling patterns of clinical observations. To fill this gap, we propose ClinPRISM, a cost-effective multimodal LLM reasoning framework for question answering over ICTS data. First, we devise an irregularity-aware multi-scale encoder to capture sparse clinical evidence at diverse temporal scales. Then, we propose a temporal evidence distiller to integrate representations across these scales and compress them into a small number of LLM-compatible tokens. Moreover, we introduce a progressive alignment strategy that sequentially aligns the irregular trajectories with the LLM's textual embedding space. To facilitate training, we construct 30,000 clinical time series paired with multi-scale descriptions, together with 41,000 instruction-tuning instances spanning 11 tasks. Using a 4-billion-parameter LLM backbone, ClinPRISM achieves state-of-the-art performance on the held-out evaluation benchmark while using only 16 time-series tokens and achieving an average inference latency of 0.15 seconds per question.
