Search papers, labs, and topics across Lattice.
This paper introduces a covering analysis framework for off-policy evaluation (OPE) in POMDPs that leverages the metric structure of the belief space to mitigate the curse of horizon and memory. By assuming Lipschitz continuity of value-relevant functions in belief space, the authors derive tighter error bounds for OPE algorithms, reducing the exponential dependence on horizon and memory length. The framework is applied to double sampling Bellman error minimization and memory-based future dependent value functions (FDVF), demonstrating improved sample efficiency through belief-space-metric-based coverage definitions.
Escape the curse of horizon and memory in offline POMDPs by exploiting the geometry of belief space, leading to tighter error bounds and improved sample efficiency in off-policy evaluation.
In off policy evaluation (OPE) for partially observable Markov decision processes (POMDPs), an agent must infer hidden states from past observations, which exacerbates both the curse of horizon and the curse of memory in existing OPE methods. This paper introduces a novel covering analysis framework that exploits the intrinsic metric structure of the belief space (distributions over latent states) to relax traditional coverage assumptions. By assuming value relevant functions are Lipschitz continuous in the belief space, we derive error bounds that mitigate exponential blow ups in horizon and memory length. Our unified analysis technique applies to a broad class of OPE algorithms, yielding concrete error bounds and coverage requirements expressed in terms of belief space metrics rather than raw history coverage. We illustrate the improved sample efficiency of this framework via case studies: the double sampling Bellman error minimization algorithm, and the memory based future dependent value functions (FDVF). In both cases, our coverage definition based on the belief space metric yields tighter bounds.
