Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper investigates the challenge of active policy selection for fine-tuning in offline-to-online reinforcement learning (O2O-RL) settings, where limited online interactions can significantly impact performance. By formulating a trade-off between evaluating multiple candidate policies and fine-tuning the most promising ones, the authors propose a novel approach that utilizes upper-confidence bounds on future performance to guide policy selection. Experimental results demonstrate that this method consistently outperforms traditional O2O-RL baselines, enhancing the efficiency of online interactions in nonstationary environments.
Actively selecting and fine-tuning policies can leverage limited online interactions to achieve superior performance in reinforcement learning, outperforming conventional methods.
Background: Offline reinforcement learning (RL) enables effective policies to be trained from large, previously collected datasets and subsequently improved through limited online interaction. This offline-to-online RL (O2O-RL) paradigm is particularly promising in nonstationary domains where interaction is costly or potentially hazardous. Standard O2O-RL pipelines train multiple candidate policies offline, evaluate them using off-policy or online evaluation, and then deploy and fine-tune the policy with the highest estimated value. However, as in offline pretraining, fine-tuning performance is highly sensitive to the choice of algorithm and hyperparameters, making it risky to commit to a single policy. Objectives: We study active policy selection for fine-tuning under a limited interaction budget in O2O-RL settings. To our knowledge, this is the first work to address this problem. Methods: We formulate the problem by identifying a fundamental trade-off between allocating online interactions to policy evaluation, which helps identify high-performing policies, and allocating them to fine-tuning, which improves policy performance. We then propose an approach that balances this trade-off by actively selecting policies for fine-tuning based on upper-confidence bounds on their future performance. These bounds are derived from locally linear performance forecasts fitted to observations obtained through online evaluation. Results: Across a diverse range of experiments, the proposed approach consistently outperforms existing O2O-RL baselines. Conclusions: Actively selecting and fine-tuning policies uses limited online interaction budgets more effectively than either committing to a single policy or dividing the budget equally among all policies. Our framework also advances offline RL toward practical deployment in real-world systems where online interaction is costly or risky.