Search papers, labs, and topics across Lattice.
This paper introduces Instruct-to-Act, a novel system that integrates vision-language models (VLMs) for high-level planning with world-model controllers for low-latency action execution. By relabeling controller policy rollouts with synthetic instructions and optimizing a behavior-cloning objective alongside traditional reward-maximizing methods, the authors enable the controllers to act autonomously based on sparse, high-level text instructions. The results demonstrate that this decoupled approach consistently outperforms both controller-only and direct VLM action-generation methods across various environments, while maintaining competitive performance against established baselines.
Instruct-to-Act reveals that decoupling planning from control can enhance action execution speed and flexibility in complex environments without sacrificing performance.
Recent work shows that pre-trained, instruction-tuned vision-language models (VLMs) perform well at mapping from instructions and observations to high-level plans, but struggle to realize such plans as reliable low-latency action sequences in unfamiliar environments. At the same time, world-model controllers excel at fast observation-to-action control, but lack open-ended task guidance. In this work, we combine these strengths into a single system, Instruct-to-Act, where we train a world-model controller to act autonomously at high frequency when conditioned on sparse, higher-latency, and high-level text instructions generated by a VLM planner. To train controllers to be language-instructable, we relabel segments of controller policy rollouts with synthetic instructions and jointly optimize a behavior-cloning objective along with existing reward-maximizing and world-modeling objectives. We evaluate our proposed approach across seven embodied environments, including three multi-agent environments where VLM planners coordinate through language while trained controllers serve as their actuators. Under matched observation and action spaces, our decoupled approach consistently outperforms controller-only and direct VLM action-generation variants, preserves fast control, and lets us swap in different pretrained VLM planners without fine-tuning, while remaining competitive with strong vision-language-action and multi-agent RL baselines on six of seven tasks.
