Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper addresses the challenge of sequentially updating policies in reinforcement learning by designing a meta-policy that plans future updates offline to balance expected cumulative value against the risk of performance regression. The authors formulate this as an optimization problem, maximizing expected cumulative value subject to a budget on the expected number of detrimental updates, and solve it using dynamic programming on a DAG representing update schedules. A key finding is that the signal-to-noise ratio of policy improvement dictates update frequency, waiting times, and risk allocation, with clearer improvements enabling more aggressive updates.
Planning policy updates offline can significantly reduce the risk of performance regression while maximizing long-term gains, even with noisy improvement signals.
Models can be retrained as new data arrive, but deploying every new version risks replacing a good policy with a worse one. We study how to plan policy updates (i.e., meta-policy) before future candidates are trained, balancing the benefits of improvement against the risk of performance regression. Our offline meta-policy maximizes expected cumulative value subject to a budget on the expected number of updates that perform worse than the policies they replace. We estimate the value and risk of possible switches from historical learning trajectories, represent an update schedule as a path in a directed acyclic graph, and select a schedule using dynamic programming. A leading-order analysis identifies the signal-to-noise ratio of policy improvement as a key driver of update frequency, waiting times, and risk allocation: clearer improvements support earlier, more frequent updates, while noisier improvements call for longer waits or greater risk expenditure. Their asymptotic rates also reveal a diminishing marginal cost of achieving greater safety over time. Experiments on synthetic and clinical trial data illustrate the performance--risk tradeoff and compare our method with alternative baselines.