Search papers, labs, and topics across Lattice.
This paper introduces a conformal policy control method that probabilistically regulates a new, potentially unsafe policy using a safe reference policy to ensure constraint satisfaction. The approach leverages conformal calibration on data from the safe policy to determine the aggressiveness of the new policy while provably enforcing a user-defined risk tolerance, without requiring accurate model identification or hyperparameter tuning. Experiments across NLP question answering and biomolecular engineering demonstrate that this method enables safe exploration from deployment and can improve performance.
Achieve safe RL exploration from day one without conservative optimization or hyperparameter tuning, by probabilistically regulating a new policy based on a safe reference policy.
An agent must try new behaviors to explore and improve. In high-stakes environments, an agent that violates safety constraints may cause harm and must be taken offline, curtailing any future interaction. Imitating old behavior is safe, but excessive conservatism discourages exploration. How much behavior change is too much? We show how to use any safe reference policy as a probabilistic regulator for any optimized but untested policy. Conformal calibration on data from the safe policy determines how aggressively the new policy can act, while provably enforcing the user's declared risk tolerance. Unlike conservative optimization methods, we do not assume the user has identified the correct model class nor tuned any hyperparameters. Unlike previous conformal methods, our theory provides finite-sample guarantees even for non-monotonic bounded constraint functions. Our experiments on applications ranging from natural language question answering to biomolecular engineering show that safe exploration is not only possible from the first moment of deployment, but can also improve performance.
