Track the frontier, from condensed matter to cosmology.
Search papers, labs, and topics across Lattice.
Role-Adaptive Policy Optimization (RAPO) addresses the mismatch in policy update needs between action selection for execution and critic bootstrapping in offline RL by introducing role-specific policy-update coefficients. These coefficients are learned by backpropagating through candidate policy updates, allowing for independent adaptation of bootstrap and execution actors in TD3+BC, and adjustment of the inverse temperature in IQL. Empirical results on D4RL locomotion and AntMaze benchmarks demonstrate RAPO's superiority over existing methods, particularly for TD3+BC.
Separating policy roles in offline RL unlocks significant performance gains by tailoring updates for value bootstrapping versus direct action execution.
Policy regularization in offline reinforcement learning balances policy improvement against reliance on uncertain value estimates. This balance can differ between selecting actions for execution and supplying actions for critic bootstrapping, yet methods such as TD3+BC couple these roles through a shared policy. We propose Role-Adaptive Policy Optimization (RAPO), which adapts policy-update coefficients according to their roles in value learning and execution. RAPO learns these coefficients by differentiating through candidate policy updates formed using the base algorithm's actor objective. For TD3+BC, RAPO separates bootstrap and execution actors and adapts their coefficients independently: the bootstrap objective penalizes policy-induced changes in target values, while the execution objective evaluates a local policy-improvement surrogate. For IQL, whose value learning is already independent of the execution actor, RAPO preserves the original value updates and adapts only the inverse temperature in advantage-weighted policy extraction. Experiments on D4RL locomotion and AntMaze tasks show improvements over both base algorithms, with larger gains for TD3+BC, whose RAPO instantiation outperforms baselines on average.