Search papers, labs, and topics across Lattice.
This paper introduces Full-Gradient Successor Feature Representations Q-Learning (FG-SFRQL), an algorithm for learning successor features (SFs) in reinforcement learning that minimizes the full Mean Squared Bellman Error. By computing gradients with respect to both online and target networks, FG-SFRQL overcomes the instability issues of semi-gradient TD methods, especially crucial for multi-task transfer. The authors theoretically prove almost-sure convergence for FG-SFRQL and empirically validate its superior sample efficiency and transfer performance against semi-gradient baselines in discrete and continuous control tasks.
Full gradient updates can stabilize successor feature learning, unlocking more effective transfer in RL.
Successor Features (SF) combined with Generalized Policy Improvement (GPI) provide a robust framework for transfer learning in Reinforcement Learning (RL) by decoupling environment dynamics from reward functions. However, standard SF learning methods typically rely on semi-gradient Temporal Difference (TD) updates. When combined with non-linear function approximation, semi-gradient methods lack robust convergence guarantees and can lead to instability, particularly in the multi-task setting where accurate feature estimation is critical for effective GPI. Inspired by Full Gradient DQN, we propose Full-Gradient Successor Feature Representations Q-Learning (FG-SFRQL), an algorithm that optimizes the successor features by minimizing the full Mean Squared Bellman Error. Unlike standard approaches, our method computes gradients with respect to parameters in both the online and target networks. We provide a theoretical proof of almost-sure convergence for FG-SFRQL and demonstrate empirically that minimizing the full residual leads to superior sample efficiency and transfer performance compared to semi-gradient baselines in both discrete and continuous domains.
