Search papers, labs, and topics across Lattice.
This paper addresses the challenge of off-policy evaluation (OPE) in finite-horizon Markov decision processes where rewards are missing not at random (MNAR), a scenario that introduces selection bias. By formalizing a reward-dependent propensity model and utilizing future states as shadow variables, the authors identify the full-data conditional mean reward without requiring explicit modeling of the MNAR mechanism. Their proposed Fitted-Q-Evaluation-style estimator demonstrates strong performance on both simulated data and MIMIC-III Sepsis data, outperforming existing methods while establishing consistency and finite-sample error bounds.
Missing rewards can skew evaluation results, but this new approach effectively mitigates selection bias in off-policy evaluation.
In offline Reinforcement Learning, immediate rewards in logged batch data are often unobserved due to sparse or irregular record-keeping, or censored beyond certain reward values. This issue arises in practical settings, including health care and marketing. We investigate off-policy evaluation (OPE) in finite-horizon Markov decision processes when rewards are missing not at random (MNAR), which breaks ignorability and induces selection bias even after conditioning on states and actions. To address this, we formalize a reward-dependent propensity model and use future states as shadow variables to identify the full-data conditional mean reward. We further introduce a bridge function that recovers the conditional mean reward without explicitly modeling the MNAR mechanism, and estimate it via a min-max procedure to avoid double sampling. Building upon these identification results, we propose an Fitted-Q-Evaluation-style estimator that propagates the recovered rewards while allowing target policies to depend on past missingness indicators. Finally, we establish consistency and finite-sample error bounds for our OPE estimator, and show through experiments the strong performance of our method compared to existing methods on simulated and MIMIC-III Sepsis data.
