Search papers, labs, and topics across Lattice.
GenerativeMPC uses a VLM-RAG module to translate visual and linguistic context into dynamic velocity limits and safety margins for a Whole-Body MPC, enabling context-aware robot control. The system also modulates virtual stiffness and damping gains for a unified impedance-admittance controller, facilitating compliant human-robot interaction. Experiments in simulation and on a physical platform demonstrate a 60% speed reduction near humans and safe, socially-aware navigation and manipulation.
Imagine robots that not only understand your instructions but also instantly adjust their movements to ensure your safety and comfort, all thanks to a VLM-RAG system that dynamically modulates MPC parameters.
Bimanual mobile manipulation requires a seamless integration between high-level semantic reasoning and safe, compliant physical interaction - a challenge that end-to-end models approach opaquely and classical controllers lack the context to address. This paper presents GenerativeMPC, a hierarchical cyber-physical framework that explicitly bridges semantic scene understanding with physical control parameters for bimanual mobile manipulators. The system utilizes a Vision-Language Model with Retrieval-Augmented Generation (VLM-RAG) to translate visual and linguistic context into grounded control constraints, specifically outputting dynamic velocity limits and safety margins for a Whole-Body Model Predictive Controller (MPC). Simultaneously, the VLM-RAG module modulates virtual stiffness and damping gains for a unified impedance-admittance controller, enabling context-aware compliance during human-robot interaction. Our framework leverages an experience-driven vector database to ensure consistent parameter grounding without retraining. Experimental results in MuJoCo, IsaacSim, and on a physical bimanual platform confirm a 60% speed reduction near humans and safe, socially-aware navigation and manipulation through semantic-to-physical parameter grounding. This work advances the field of human-centric cybernetics by grounding large-scale cognitive models into predictable, high-frequency physical control loops.
