Search papers, labs, and topics across Lattice.
This paper introduces a framework called Conservative Query and Adaptive Regularization that enhances offline reinforcement learning (RL) by addressing the challenges of selecting informative preference queries and effectively utilizing the feedback from those queries. By employing a Morse network for uncertainty estimation, the authors develop a conservative query strategy that stabilizes policy updates while maintaining proximity to the dataset, alongside an adaptive regularization scheme that adjusts constraints dynamically during optimization. Experimental evaluations on the D4RL benchmark show that this approach achieves superior or competitive performance across various tasks, highlighting its effectiveness in offline RL settings.
Uncertainty-aware querying can significantly stabilize policy updates in offline reinforcement learning, leading to improved performance without additional environment interaction.
Offline reinforcement learning (RL) aims to learn an effective policy from a static dataset, but its performance is fundamentally limited by dataset coverage. Action preference queries leverage expert feedback without additional environment interaction, enabling policy improvement during offline training. However, existing methods still face two key challenges: selecting informative preference queries and effectively exploiting the collected feedback. Current approaches typically rely only on the distance between policy actions and dataset actions for query selection, while enforcing fixed constraints that keep the policy close to queried preferences. Such strategies often lead to unstable policy updates and integrate poorly with value regularization. To address these limitations, we propose Conservative Query and Adaptive Regularization under Uncertainty Estimation, a lightweight framework that jointly improves preference querying and preference exploitation. Specifically, we employ a Morse network to estimate the uncertainty of policy actions with respect to the offline dataset. Based on this uncertainty, we introduce a conservative query strategy that selectively queries actions near the dataset to preserve Bellman-update stability, together with an uncertainty-aware adaptive regularization scheme that dynamically adjusts data-level constraints during policy optimization. We integrate our framework with CQL and evaluate it extensively on the D4RL benchmark. Experimental results demonstrate superior or competitive performance across a wide range of tasks.
