Search papers, labs, and topics across Lattice.
This paper introduces GAINS, a novel framework that addresses the challenges posed by inconsistent human intervention signals in reinforcement learning (RL) for robot manipulation. By employing distributional RL with quantile Q-networks, GAINS effectively models the variability in returns caused by sparse rewards and imperfect human corrections, leading to a pessimistic exploration strategy that enhances safety and sample efficiency. The framework demonstrates significant improvements in task success rates and recovery in failure scenarios, achieving a 22% higher success rate compared to existing methods in simulated and real-world tasks.
GAINS reveals that effectively modeling human imperfection can boost task success rates by over 20% in robot manipulation tasks.
Correcting robot manipulation policies through human intervention holds great promise for real-world deployment, yet human operators are inherently imperfect in both the actions they provide and the timing of their intervention signals. While the former has been extensively discussed in reinforcement learning (RL), the latter remains underexplored. At high control frequencies, human intervention signals are often delayed and inconsistent across time and state space. In this work, we present GAINS, a framework for leveraging inconsistent human intervention signals in RL. At the core of GAINS, we employ distributional RL with quantile Q-networks to model the return variability induced by sparse task rewards and inconsistent human interventions. Building on this distributional representation, we introduce a pessimistic exploration strategy that promotes safe and sample-efficient learning under human corrections. We evaluate GAINS on four diverse simulated manipulation tasks and two challenging real-world scenarios against state-of-the-art intervention-based methods. GAINS achieves a 22% higher task success rate than RLIF and improves recovery success by up to 43% in failure scenarios. These results highlight the importance of modeling return variability induced by human imperfection for real-world deployment of intervention-based learning.
