Search papers, labs, and topics across Lattice.
This paper introduces a language-critique framework for imitation learning that utilizes natural language as a structured supervision signal, moving beyond traditional scalar feedback methods. By generating language labels that articulate task progress, identify suboptimal behaviors, and offer corrective guidance, the authors create a language-critique loss that trains policies more effectively. Empirical evaluations across various continuous control tasks show that this approach consistently outperforms existing imitation learning and offline reinforcement learning baselines, highlighting the potential of language in enhancing policy learning from suboptimal demonstrations.
Language can be a powerful supervisory tool, enabling imitation learning to outperform traditional methods by providing structured feedback on task performance.
Prior work on imitation learning from suboptimal demonstrations typically relies on compressed supervision signals such as confidence estimates, discriminator scores, or importance weights. These scalar signals are inherently limited, as they cannot explicitly express intermediate reasoning about task progress, failure modes, or corrective actions. We propose a language-critique framework for imitation learning from suboptimal demonstrations that instead leverages natural language as a structured supervision signal, avoiding the collapse of expressive feedback into scalars. Our method first constructs language labels from demonstrations that explicitly describe current progress, identify suboptimal behaviors, and provide fine-grained corrective guidance. We then introduce a language-critique loss that directly trains policies using these structured signals without reducing them to scalars, and instantiate it for both behavior cloning and diffusion policies, yielding LC-BC and LC-DP. We further provide a theoretical result showing that the proposed objective upper-bounds the expert performance gap under standard assumptions. Empirically, we evaluate on diverse continuous control tasks spanning navigation, manipulation, and gameplay, where our methods consistently outperform strong imitation learning and offline reinforcement learning baselines. These results demonstrate that language can serve as a powerful and structured form of supervision for learning robust policies from suboptimal data.
