Search papers, labs, and topics across Lattice.
This paper introduces Hierarchical Implicit Q-Chunking (HiQC), an offline goal-conditioned reinforcement learning algorithm that addresses the challenges of long-horizon tasks by integrating high-level latent planning with low-level action chunking. By conditioning the low-level critic on temporally extended action sequences, HiQC achieves unbiased k-step value backups, effectively compressing the horizon during both planning and execution. Theoretical analysis shows that this approach results in a tighter bound on value error, and empirical results demonstrate HiQC's superior performance on long-horizon navigation tasks in the OGBench suite, particularly in humanoid-giant scenarios.
HiQC achieves unprecedented performance on long-horizon tasks by combining high-level planning with low-level action chunking, effectively reducing value estimation errors.
Offline goal-conditioned reinforcement learning (RL) holds the promise of learning general-purpose policies from static datasets. However, scaling these methods to long-horizon tasks remains a challenge due to the curse of horizon, where value estimation errors can compound through long chains of bootstrapped Bellman backups. Existing hierarchical approaches mitigate this by decomposing tasks into subgoals, yet they often rely on low-level controllers that suffer from myopic execution and biased value estimates. In this work, we propose Hierarchical Implicit Q-Chunking (HiQC), an offline goal-conditioned RL algorithm that combines high-level latent planning with low-level action chunking. By conditioning the low-level critic on temporally extended action sequences, HiQC enables unbiased k-step value backups, compressing the horizon at both the planning and execution levels. We theoretically demonstrate that this dual decomposition results in a tighter bound on value error under a bounded per-backup error model compared to standard hierarchy or flat chunking alone. Empirically, HiQC achieves the highest aggregate performance among the compared methods on the OGBench suite, with its largest gains on long-horizon navigation tasks such as humanoid-giant.
