Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces Outcome-Predictive State Representations (OPSRs), a task-independent state abstraction based on predicting environment outcomes from the agent's perspective. They demonstrate that OPSRs enable optimal but limited transfer, and address this limitation by incorporating OPSR-based skills (abstract actions learned from options). Empirical results show that learning OPSR-based skills from demonstrations significantly accelerates learning in novel tasks without pre-processing.
Forget task-specific features: reusable skills built on outcome prediction can unlock faster learning in unseen RL environments.
A key challenge in scaling up Reinforcement Learning is generalizing learned behaviour. Without the ability to carry forward acquired knowledge an agent is doomed to learn each task from scratch. In this paper we develop a new formalism for transfer by virtue of state abstraction. Based on task-independent, compact observations (outcomes) of the environment, we introduce Outcome-Predictive State Representations (OPSRs), agent-centered and task-independent abstractions that are made up of predictions of outcomes. We show formally and empirically that they have the potential for optimal but limited transfer, then overcome this trade-off by introducing OPSR-based skills, i.e. abstract actions (based on options) that can be reused between tasks as a result of state abstraction. In a series of empirical studies, we learn OPSR-based skills from demonstrations and show how they speed up learning considerably in entirely new and unseen tasks without any pre-processing. We believe that the framework introduced in this work is a promising step towards transfer in RL in general, and towards transfer through combining state and action abstraction specifically.