Track synthesis, materials and catalysis as it is published.
Search papers, labs, and topics across Lattice.
This paper investigates augmenting Vision-Language-Action (VLA) models with non-linguistic conditioning signals, specifically electromyography (EMG) envelopes and visual segmentation annotations, to improve performance in ambiguous environments. By integrating these modalities into EC-VLA and VA-VLA architectures respectively, the researchers demonstrate significant performance gains in cluttered, out-of-distribution scenarios compared to language-only baselines. The findings underscore the value of multimodal state conditioning for enhancing VLA robustness and generalization.
Electromyography and visual segmentation annotations unlock substantial VLA performance gains in cluttered, out-of-distribution task conditions, outperforming language-only conditioning.
Vision-Language-Action (VLA) models rely strongly on language for describing task information, despite having multimodal inputs. We hypothesize that other modalities in the state space may present opportunities for supplemental task conditioning, which may be particularly relevant in cluttered or otherwise ambiguous scenes. We introduce two tuned models to test this hypothesis: (1) an electrophysiology-conditioned VLA (EC-VLA) that incorporates 8-channel electromyography envelopes as continuous conditioning input concatenated to the proprioceptive vector, and (2) a visually-annotated VLA (VA-VLA) that incorporates visual segmentation annotations to the image inputs. On a cube-selection task evaluated across three participants, EC-VLA matches a language-prompted baseline in uncluttered, in-distribution conditions and substantially outperforms it in cluttered, out-of-distribution scenes. Similarly, VA-VLA shows modest improvements over a language-prompted baseline in in-distribution scenes with substantial improvement in cluttered, out-of-distribution trials. Together, these results provide strong evidence for the potential benefit of task-conditioning beyond language.
Johns Hopkins