Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper addresses the challenge of offline reinforcement learning (RL) when true actions are unobserved, introducing an influence-function-based estimator called LURE (Learning from the Unseen: Robust Estimator) that leverages next-state variables as proxies for hidden actions. The authors demonstrate that LURE is multiply robust and maintains consistency under various nuisance component specifications, enabling valid statistical inference. Through simulations and a practical application in sepsis management using the MIMIC-III database, the effectiveness of LURE is validated, marking a significant advancement in offline RL methodologies.
Hidden actions in offline RL can lead to biased outcomes, but LURE offers a robust solution that ensures reliable policy evaluation.
Standard offline reinforcement learning (RL) algorithms typically assume that the actions in the dataset are observed without error. However, in many real-world applications, the true actions are unobserved and only noisy proxies are available, causing existing RL methods to yield biased and potentially misleading conclusions. We study off-policy evaluation in infinite-horizon discounted Markov decision processes with hidden actions. By leveraging the next-state variable as a natural proxy for the unobserved action, we establish identification of the policy value and propose an influence-function-based estimator called LURE (Learning from the Unseen: Robust Estimator). LURE is multiply robust, remaining consistent under several combinations of correctly specified nuisance components, and is asymptotically normal, enabling valid statistical inference. To our knowledge, this is the first work to address offline RL with hidden actions. We demonstrate LURE's effectiveness through simulations and a sepsis management application using the MIMIC-III database.