Search papers, labs, and topics across Lattice.
This paper introduces IMCopilot, a set of RL-trained atomic skills for shared-autonomy teleoperation data collection and low-level execution, and MoDE-VLA, a VLA architecture that integrates force and tactile modalities via residual injection. IMCopilot simplifies the acquisition of high-fidelity data for bimanual dexterous manipulation, while MoDE-VLA refines actions based on contact awareness without sacrificing pre-trained knowledge. Experiments on four tasks show a doubled success rate improvement over baseline methods in dexterous contact-rich tasks.
Unlock human-like dexterity in robotic manipulation by combining RL-assisted teleoperation with a novel VLA architecture that leverages force and tactile feedback.
While Vision-Language-Action (VLA) models have demonstrated remarkable success in robotic manipulation, their application has largely been confined to low-degree-of-freedom end-effectors performing simple, vision-guided pick-and-place tasks. Extending these models to human-like, bimanual dexterous manipulation-specifically contact-rich in-hand operations-introduces critical challenges in high-fidelity data acquisition, multi-skill learning, and multimodal sensory fusion. In this paper, we propose an integrated framework to address these bottlenecks, built upon two components. First, we introduce IMCopilot (In-hand Manipulation Copilot), a suite of reinforcement learning-trained atomic skills that plays a dual role: it acts as a shared-autonomy assistant to simplify teleoperation data collection, and it serves as a callable low-level execution primitive for the VLA. Second, we present MoDE-VLA (Mixture-of-Dexterous-Experts VLA), an architecture that seamlessly integrates heterogeneous force and tactile modalities into a pretrained VLA backbone. By utilizing a residual injection mechanism, MoDE-VLA enables contact-aware refinement without degrading the model's pretrained knowledge. We validate our approach on four tasks of escalating complexity, demonstrating doubled success rate improvement over the baseline in dexterous contact-rich tasks.
