Search papers, labs, and topics across Lattice.
MuseVLA is an adaptive multimodal sensing Vision-Language-Action model designed for robotic manipulation that integrates diverse sensor modalities beyond RGB observations. By generating sensor tokens and target descriptions, it effectively selects and utilizes various sensing modalities, allowing for enhanced perception of physical properties like temperature and sound. Evaluated on complex dexterous tasks, MuseVLA achieved an average success rate of 80.6%, significantly outperforming traditional RGB-only and multisensory VLA baselines while demonstrating robust zero-shot performance on unseen tasks.
MuseVLA achieves an impressive 80.6% success rate in robotic manipulation tasks by leveraging diverse sensing modalities, surpassing traditional RGB-only models.
Humans naturally leverage diverse sensing modalities to interact with the physical world, while most Vision-Language-Action (VLA) models for robotics rely solely on RGB observations. This limits their ability to perceive physical properties that are difficult or impossible to infer from RGB cameras, such as temperature, sound, or radar response. We present MuseVLA, an adaptive multimodal sensing VLA model that integrates novel sensors as on-demand tools for robotic manipulation. Given a task instruction and visual context, MuseVLA first generates a sensor token and target description that select the sensing modality to invoke and what to attend to, analogous to a tool call with arguments. It then converts the selected sensor measurement into a grounded sensor image, a unified intermediate representation that encodes heterogeneous readings for multimodal fusion and action generation. This design decouples sensor-specific processing from the VLA backbone, enabling efficient integration of diverse modalities. To reduce the need for expensive multisensory robot datasets, we further introduce a data synthesis pipeline that augments existing RGB video datasets with grounded sensor images, enabling generalization to unseen sensor-guided tasks. We evaluate MuseVLA on a real-world robot across challenging dexterous hand manipulation tasks that require multimodal sensing inputs, including temperature-guided pick-and-place, audio-driven object search, and radar-assisted hidden object retrieval. MuseVLA achieves 80.6% success rate on average, outperforming RGB-only and multisensory VLA baselines significantly, and exhibits strong zero-shot capabilities on unseen tasks.
Microsoft Research
Tsinghua AI