Search papers, labs, and topics across Lattice.
This paper introduces neurosymbolic hierarchical reinforcement learning (HRL) that integrates Incremental Knowledge (InK) to enhance sample efficiency in environments with sparse rewards. By allowing symbolic high-level components to perform planning on an updatable representation of knowledge, and enabling low-level neural modules to learn from experience, the proposed method significantly improves decision-making in long-horizon tasks. Experimental results in navigation tasks show that InK leads to a marked increase in sample efficiency compared to traditional fixed HRL approaches.
Incorporating incremental knowledge into hierarchical reinforcement learning can drastically enhance sample efficiency in challenging environments with sparse rewards.
(Flat) Reinforcement Learning (RL) agents face significant challenges in environments with sparse rewards that require long-horizon reasoning. A compelling approach to improve sample efficiency is to incorporate knowledge into learning and decision-making. In standard Hierarchical RL (HRL), knowledge is encoded in a fixed, non-updatable form, such as architectural choices, and remains unchanged throughout learning. With fixed HRL, reasoning with incremental knowledge learned during exploration is impractical before sufficient environmental knowledge is acquired, leading to poor sample efficiency. In this work, we propose neurosymbolic HRL with {\em Incremental Knowledge (InK)}: symbolic high-level components perform {\em symbolic planning} (e.g. using $D^*$) on an updatable representation of current InK, while low-level goal-conditioned neural modules learn motion primitives through experience using reward shaping. Experiments on navigation tasks demonstrate that incorporating InK substantially improves sample efficiency. Additionally, to perform {\em optimal} symbolic planning given {\em prior} knowledge about the world, we develop Belief World Tree Search. The code is available at https://github.com/CPS-research-group/ink_bwts.
