Search papers, labs, and topics across Lattice.
This paper introduces flow control for vision-language-action (VLA) models, allowing real-time steering of actions using simple, generic inputs like a keyboard without the need for retraining. The method effectively translates crude user inputs into high-quality action samples that align with the learned expert action distribution, enhancing both fidelity to user intent and overall action quality. Results show that flow control significantly improves task success rates and completion speed, while also benefiting from fine-tuning on flow control trajectories to enhance the autonomous policy further.
Users can now steer vision-language-action models in real-time with minimal input, drastically improving task performance without complex retraining.
We introduce flow control of vision-language-action (VLA) models, a simple and effective way to steer VLA actions in real-time through generic inputs, such as a keyboard. This method can be used out-of-the-box and does not require retraining or fine-tuning VLAs. It enables relatively crude user inputs to steer a VLA to align with user intent. The VLA transforms these inputs into action samples drawn from the VLA expert action distribution learned during training, so that the generated actions are high quality (conformity to the action expert distribution) and high fidelity (reflecting the user's intent). We demonstrate that flow control has many desirable properties: (1) flow control accurately and responsively steers robot actions with user inputs, (2) it is robust to suboptimal user inputs, (3) it enables users to steer VLAs to achieve significantly higher success rates and faster task completion, and (4) fine-tuning a VLA on flow control trajectories improves the autonomous policy. Together, these results provide a simple and intuitive way for users to help steer VLA actions, increasing task performance.
