Search papers, labs, and topics across Lattice.
This paper introduces the TOUR benchmark, which addresses the challenges of trajectory-level unlearning in offline reinforcement learning by evaluating the effectiveness of data deletion methods. The authors demonstrate that common deletion techniques exhibit varying privacy-utility trade-offs depending on the environment and the specific auditing methods used. Key findings reveal that retraining and fine-tuning often outperform standard deletion baselines in retaining utility, highlighting the complexity of ensuring effective unlearning in RL systems.
Unlearning in offline RL is more complex than previously thought, with common deletion methods showing environment-dependent privacy-utility behavior that can mislead evaluations.
Offline Reinforcement Learning (RL) agents are trained on fixed behavioral trajectories, which makes trajectory-level deletion important when selected data must be removed after training. Evaluating such deletion is difficult because a lower membership score can reflect trajectory removal, residual memorization visible to another attack, or policy collapse that destroys useful behavior. We introduce Trajectory-level memOrization and Unlearning in offline RL (TOUR), a benchmark that combines trajectory-level partitioning, matched non-member controls, retraining references, retained-performance anchors, and multi-attack privacy auditing. Across D4RL locomotion experiments and an exploratory AntMaze extension, TOUR shows that common deletion baselines have environment-dependent privacy-utility behavior. Retraining and fine-tuning often provide stronger retained-utility references than uniform GA+Refit, while TrajDeleter remains a useful comparator but is not uniformly stronger under the same audit. Reference-model, threshold, deviation, equivalence, action-error, representation-based, and query-limited attacks further show that a single likelihood-based membership score can overstate deletion quality. In the evaluated settings, conclusions about offline RL unlearning are therefore not stable under single-score auditing. They depend on matched non-member construction, retraining-relative calibration, attack family, retained utility, and explicit scope for diagnostic architecture or component-level evidence.
