Search papers, labs, and topics across Lattice.
This paper introduces a novel approach to probabilistic shielding in safe reinforcement learning by integrating online model learning with adaptive shield computation. By estimating transition probabilities as the RL agent explores its environment, the method allows for a dynamic shield that evolves alongside the agent's learning process. Empirical evaluations demonstrate that this adaptive shielding can effectively balance exploration and safety, addressing key challenges in environments where MDP transition probabilities are initially unknown.
Adaptive probabilistic shielding can evolve in real-time, enhancing safety in reinforcement learning without sacrificing exploration.
Probabilistic shielding is a technique for safe reinforcement learning (RL). Typically, a static observer -- called the shield -- constrains the learning agent's actions to those for which acting safely remains feasible. Traditionally, the shield is computed from the transition probabilities of the underlying Markov decision process (MDP). Thus, this technique is not applicable when the MDP model is not given a priori, which, unfortunately, is the case in typical RL applications. In this paper, we study the problem of computing a shield in the setting where the transition graph of the MDP is known, but the transition probabilities are unknown. Our approach integrates probabilistic shielding with online model learning: as the RL agent explores the environment, we estimate the transition probabilities. From this estimate, we compute a shield. While the shield may be conservative initially, it adapts as the model estimate becomes more precise. Thus, the shield improves in tandem with the RL agent. This paradigm of adaptive probabilistic shielding raises a number of challenges, such as when to recompute the shield and how to balance between exploration and safety during learning. We empirically evaluate multiple variants of this paradigm across several environments.
