Search papers, labs, and topics across Lattice.
The paper introduces DM-Bench, a novel benchmark for evaluating LLMs on personalized decision-making tasks relevant to diabetes management, focusing on patient-facing AI solutions. DM-Bench uses a dataset of CGM data and behavioral logs from 15,000 individuals to generate 360,600 personalized questions across seven task categories. Evaluating eight recent LLMs reveals significant performance variability across tasks and metrics (accuracy, groundedness, safety, clarity, and actionability), highlighting the need for specialized benchmarks in this domain.
LLMs struggle with the nuances of personalized diabetes management, as evidenced by inconsistent performance across a new benchmark spanning glucose interpretation, behavioral associations, and long-term planning.
We present DM-Bench, the first benchmark designed to evaluate large language model (LLM) performance across real-world decision-making tasks faced by individuals managing diabetes in their daily lives. Unlike prior health benchmarks that are either generic, clinician-facing or focused on clinical tasks (e.g., diagnosis, triage), DM-Bench introduces a comprehensive evaluation framework tailored to the unique challenges of prototyping patient-facing AI solutions in diabetes, glucose management, metabolic health and related domains. Our benchmark encompasses 7 distinct task categories, reflecting the breadth of real-world questions individuals with diabetes ask, including basic glucose interpretation, educational queries, behavioral associations, advanced decision making and long term planning. Towards this end, we compile a rich dataset comprising one month of time-series data encompassing glucose traces and metrics from continuous glucose monitors (CGMs) and behavioral logs (e.g., eating and activity patterns) from 15,000 individuals across three different diabetes populations (type 1, type 2, pre-diabetes/general health and wellness). Using this data, we generate a total of 360,600 personalized, contextual questions across the 7 tasks. We evaluate model performance on these tasks across 5 metrics: accuracy, groundedness, safety, clarity and actionability. Our analysis of 8 recent LLMs reveals substantial variability across tasks and metrics; no single model consistently outperforms others across all dimensions. By establishing this benchmark, we aim to advance the reliability, safety, effectiveness and practical utility of AI solutions in diabetes care.
