Search papers, labs, and topics across Lattice.
This paper introduces OpenTumorBoard, a novel benchmark comprising 611 real-world patient cases and 19,157 discussion turns from multidisciplinary tumor boards, designed to evaluate LLMs in both single-turn specialist responses and full board simulations. Despite extensive evaluation of 14 frontier and medical LLMs, current models exhibit significant limitations, achieving only moderate clinical equivalence and alignment with recorded board conclusions. Supervised finetuning and RL demonstrate potential for adaptation, with expert review confirming the benchmark's high information coverage and fidelity.
Frontier LLMs struggle to replicate expert consensus in complex, longitudinal cancer treatment discussions, scoring below expert equivalence on a new real-world benchmark.
Multidisciplinary tumor boards integrate multimodal clinical observations and longitudinal patient histories through specialist discussions, yet benchmarks rarely capture these real-world trajectories. We introduce OpenTumorBoard, a benchmark with 611 patient cases and 19,157 discussion turns across ten specialist roles, transcribed from 12,534 minutes of publicly available tumor board recordings on YouTube. The benchmark evaluates two settings: SPECIALIST TURN, in which an LLM responds to a clinically significant question posed during a real discussion, and BOARD SIMULATION, in which it generates an entire back-and-forth discussion and reaches a consensus on therapy recommendations, surgical plans, next actions and clinical trial matching. Evaluation of 14 general-purpose frontier and medical LLMs reveals substantial limitations: the best models score 3.43 out of 5 in clinical equivalence to specialist answers and 2.78 out of 5 in alignment with recorded board conclusions. Supervised finetuning and reinforcement learning improve performance on a held-out test set, suggesting that real-world discussion trajectories can support model adaptation. Three M.D. experts review a subset of the benchmark, finding high information coverage and factuality of patient cases and strong fidelity of extracted consensus conclusions. We will release OpenTumorBoard and its automated curation pipeline to support the development and evaluation of LLMs for multidisciplinary, personalized cancer decision-making.
Microsoft Research
UW