Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
SHERPA alternates control between a heuristic policy and an RL policy in contiguous segments to improve learning in sparse-reward, long-horizon tasks. This segmented approach preserves coherent sub-trajectories and enables the RL policy to surpass the heuristic guidance through RL optimization. Experiments across ten simulated tasks and real-world robotic manipulation demonstrate that SHERPA outperforms RL, IL, and other heuristic-guided baselines, even with degraded heuristics.
Heuristic guidance doesn't have to be a crutch: SHERPA shows how to leverage heuristics for faster RL, then surpass them.
In sparse-reward, long-horizon domains, reinforcement learning (RL) often suffers from slow convergence and instability, complicating robotic manipulation. Previous heuristic-guided approaches have relied on step-level actions and imitation loss, but struggle to maintain temporal coherence or solve multi-stage tasks. We present SHERPA (Segmented Heuristic Experience Relay for Policy Assistance), which alternates control between a heuristic policy and an RL policy in contiguous segments, preserving coherent sub-trajectories and yielding more stable learning. Unlike imitation-based designs, SHERPA learns entirely through RL optimization, enabling the policy to benefit from and surpass heuristic guidance. We also incorporate phase-specific rewards derived from heuristic rules to provide targeted subgoal feedback. Across ten tasks in the Fetch and Panda suites, including four long-horizon benchmarks, SHERPA consistently outperforms RL, IL, and heuristic-guided baselines and demonstrates robustness even under degraded heuristics. Real-world experiments on a UR5 robot further confirm SHERPA’s practical scalability.