Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces a teacher-student representational alignment method for imitation learning, addressing the imitation gap caused by the teacher's reliance on privileged state information unavailable to the student. The approach learns a shared embedding space via contrastive learning, simultaneously training the teacher policy while preventing the extraction of private information by limiting gradients to the encoder networks. Experiments across several domains demonstrate that this method achieves higher student performance and reduces the imitation gap compared to state-of-the-art baselines.
Force your teacher to see the world as the student does, and imitation learning suddenly works much better.
Imitation learning (IL) from a state-based reinforcement learning (RL) policy is a common approach to overcome the curse of dimensionality in complex and high-dimensional observation spaces prevalent in robotics. This paper addresses the irreducible imitation gap that emerges when teacher and student are learned in isolation, and the teacher policy has the liberty to rely on privileged state information that the student cannot infer from its observations. Instead of improving poor student performance with RL finetuning after IL, which often requires a whole new training setup, we propose a novel algorithm which learns a shared embedding space that hides agent-specific observations and thus trains imitable teacher policies by construction. We train the shared embedding space with self-supervised contrastive learning in parallel to the teacher policy and prevent it from extracting private information by limiting its gradients from updating the encoder networks. We perform evaluations on several example domains and compare to state-of-the-art baselines showing that our algorithm enables higher student performance with substantially reduced imitation gap.