Search papers, labs, and topics across Lattice.
This paper introduces Direction-Conditioned Policies (DCP), an online method for goal-conditioned reinforcement learning that decomposes goal-reaching into subgoal scoring and direction conditioning, addressing the limitations of standard approaches that rely on raw goal signals. By leveraging Hamilton-Jacobi-Bellman theory, the authors demonstrate that optimal actions depend on the gradient of the goal-reaching distance, leading to improved performance across nine environments, particularly in manipulation and obstacle-interaction tasks. The results highlight that DCP not only enhances learning efficiency but also aligns the actor's conditioning input during training and deployment, achieving significant gains over Contrastive RL.
Direction-Conditioned Policies achieve substantial performance improvements in goal-conditioned reinforcement learning by effectively utilizing gradient information, outperforming existing methods in complex environments.
Hamilton-Jacobi-Bellman theory implies that the optimal goal-conditioned action depends on the goal only through the gradient of the goal-reaching distance at the current state, yet standard online GCRL still conditions the actor on the raw goal -- a signal that is geometrically uninformative when the goal is far from the data distribution. We propose Direction-Conditioned Policies (DCP), a fully online method that decomposes goal-reaching into two components sharing one InfoNCE representation $蠄$: a subgoal-scoring step that selects a visited state $z_t$ aligned with the final goal $g$ in $蠄_g$, and a direction-conditioned actor that consumes the unit direction $d_t$ and magnitude $r_t$ from $蠄(s_t)$ to $蠄(z_t)$. The two components train jointly, factor cleanly at deployment (subgoal scoring is removed, while direction conditioning remains with $g$ in place of $z_t$), and admit independent modification at the same $(d_t,r_t)$ interface. We prove three results. First, direction sufficiency under HJB: the optimal action under control-affine dynamics depends on the goal only through the value gradient. Second, a quantitative bound showing that, under mild conditions on the learned representation and assuming the scoring rule returns an on-path $z_t$, the actor's conditioning input at training and at deployment coincide up to representation error and geodesic slack. Third, a controllable-subspace characterization of when directional conditioning fails. Across nine environments, DCP improves over Contrastive RL on most final metrics, with the largest gains on manipulation and obstacle-interaction tasks; a qualitative analysis of the learned $蠄$-distance landscape shows the contrastive representation behaves as an online quasimetric encoding environment topology, and the single failure case (AntSoccer) localizes to a learned-gradient pathology that the theory anticipates.
