Search papers, labs, and topics across Lattice.
This paper addresses the problem of cross-domain offline reinforcement learning (CDRL), where the goal is to improve policy learning in a target domain using data from a different source domain. The key idea is to assess the transferability of source-domain data based on its alignment with target-domain Bellman targets, rather than relying on transition-level similarity. The proposed Target-Aligned Bellman Backup (TABB) method selectively leverages source-domain data based on its contribution to accurate Bellman target estimation, leading to improved performance in CDRL settings with limited target-domain data.
Source data that looks similar can still tank your cross-domain RL: aligning with target-domain Bellman targets is what actually matters for transfer.
Cross-domain offline reinforcement learning (CDRL) aims to improve policy learning in a target domain by leveraging data collected from a source domain. Existing works typically assess the transferability of source-domain data by measuring its similarity to target-domain transitions, and implicitly perform transition-level selection. Transitions that are considered similar are assigned higher weights or rewards, while dissimilar ones are down-weighted. However, transition-level similarity does not necessarily imply consistency in long-term returns. Even visually or dynamically similar transitions may lead to significantly different outcomes in the target domain, which can mislead policy learning and degrade performance. To address this issue, we revisit the fundamental objective of policy learning. Since policy optimization ultimately relies on Bellman targets to evaluate the quality of decisions, we propose to assess the transferability of source-domain transitions based on their alignment with target-domain Bellman targets, rather than superficial transition similarity. Based on this insight, we propose a method termed Target-Aligned Bellman Backup (TABB), which selectively leverages source-domain data by measuring their contribution to accurate Bellman target estimation in the target domain. We evaluate TABB across a broad range of cross-domain offline RL settings with highly limited target-domain data. Experimental results show that TABB consistently achieves strong performance.
