Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces a method for inferring reward machines directly from raw state trajectories without access to rewards, labels, or machine nodes. They identify the trajectory data sufficient for learning in this information-scarce regime and extend the approach to an active learning setting where trajectory extensions are queried to improve data efficiency. Experiments on grid world examples demonstrate the feasibility of learning reward machines from raw state data using active querying.
Forget hand-engineering reward machines: this work learns them directly from raw state trajectories, even without reward or label information.
Reward machines are automaton-like structures that capture the memory required to accomplish a multi-stage task. When combined with reinforcement learning or optimal control methods, they can be used to synthesize robot policies to achieve such tasks. However, specifying a reward machine by hand, including a labeling function capturing high-level features that the decisions are based on, can be a daunting task. This paper deals with the problem of learning reward machines directly from raw state and policy information. As opposed to existing works, we assume no access to observations of rewards, labels, or machine nodes, and show what trajectory data is sufficient for learning the reward machine in this information-scarce regime. We then extend the result to an active learning setting where we incrementally query trajectory extensions to improve data (and indirectly computational) efficiency. Results are demonstrated with several grid world examples.