Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This study systematically evaluates five offline reinforcement learning (RL) algorithm families and 14 reward designs using a large dataset of 44,894 acute ischemic stroke patients to assess the effectiveness of RL policies compared to physician decisions. The findings reveal that while initial estimates suggest a modest improvement in clinical outcomes, confounding factors embedded in reward structures significantly inflate these estimates, leading to misleading conclusions about the efficacy of RL approaches. After applying deconfounding techniques, the apparent policy improvement diminishes substantially, indicating that previous claims of RL superiority may be overstated and emphasizing the need for rigorous evaluation methods in clinical settings.
Reward confounding in offline RL for stroke treatment inflates perceived policy improvements by over 200%, challenging the validity of previous claims about RL outperforming physician decisions.
Recent offline reinforcement learning (RL) studies report policies that outperform physician decisions on clinical outcomes. We conduct a systematic, partially crossed evaluation of five offline RL algorithm families and 14 reward designs in 44,894 post-2018 acute ischemic stroke patients from a nationwide registry (N = 129,033). Standard Fitted Q-Evaluation (FQE) yields an apparent policy-improvement estimate of +0.0069; adding an Early Neurological Deterioration penalty increases it to +0.0101. We identify reward-embedded confounding, in which a proxy terminal reward encodes baseline severity and prognosis as well as treatment efficacy. A 2 x 2 factorial analysis finds that terminal reward confounding accounts for 218.6% of the observed signal change, so its removal overshoots the null. After DML-inspired GBM reward residualization, the FQE estimate attenuates to +0.0033 (p = 0.132), and full deconfounding yields +0.0025 (p = 0.291). FQE-based diagnostics, T-learner analyses, and direct recurrence analyses converge away from a clinically meaningful aggregate improvement. A 1-year mRS factorial analysis replicates the attenuation. We provide an empirically motivated six-step evaluation checklist. NIHSS-stratified heterogeneity is hypothesis-generating for prospective trial design; hospital-level disagreement does not persist after full reward deconfounding.