Search papers, labs, and topics across Lattice.
The paper introduces MIMIC, a framework for imitation learning that leverages language as an internal representation of behavioral intent, inspired by the cognitive role of inner speech. MIMIC uses a vision-language model to train a conditional variational autoencoder that generates inner speech from observations, which is then used to condition a diffusion-based behavior cloning policy. Experiments show that MIMIC improves behavior diversity, fidelity to human demonstrations, and enables fine-grained behavioral steering in robotic manipulation and human-AI collaboration tasks.
Unlock nuanced control over imitation learning agents by steering their "inner speech," enabling diverse and faithful behavior without retraining.
Effective human-AI coordination requires artificial agents capable of exhibiting and responding to human-like behaviors while adapting to changing contexts. Imitation learning has emerged as one of the prominent approaches to build such agents by training them to mimic human-demonstrated behaviors. However, current methods struggle to capture the inherent diversity and non-Markovian nature of human behavior and lack the ability to steer behavior at inference time. Drawing inspiration from the theory of human cognitive processes, where inner speech guides action selection before execution, we propose MIMIC (Modeling Inner Motivations for Imitation and Control), a framework that uses language as an internal representation of behavioral intent. MIMIC employs the novel use of vision-language models as linguistic scaffolding to train a conditional variational autoencoder capable of generating inner speech from observations. A diffusion-based behavior cloning policy then selects actions conditioned on current observations and the generated inner speech. MIMIC enables fine-grained steering of behavior at inference time by conditioning the agent on behavior-specific speech. Experiments across robotic manipulation tasks and human-AI collaboration games demonstrate that MIMIC significantly enhances both behavior diversity and fidelity to human demonstrations while enabling nuanced behavioral steering without training on additional demonstrations. We open source our code and provide pre-trained MIMIC agents and qualitative demos at: https://mimic-research.github.io.
