Search papers, labs, and topics across Lattice.
This paper introduces FWBC-VLA, a novel framework that integrates force awareness into vision-language-action (VLA) models for wheeled-legged robots engaged in contact-rich loco-manipulation tasks. By employing a sensorless residual-torque estimator, HSR-Force, the framework effectively infers contact strength and its variations, allowing for enhanced task execution without the need for additional hardware. Real-world experiments, including whiteboard wiping and door opening, validate that FWBC-VLA significantly improves the robot's ability to manage physical interactions during manipulation tasks.
Robots can now perform complex manipulation tasks more effectively by understanding and responding to contact forces without additional sensors.
Contact-rich loco-manipulation requires a bridge between semantic action generation and physical interaction control. Existing Vision-language-action (VLA) models generate task-level actions from visual and linguistic observations, but cannot interpret the physical interactions induced by those actions. While the whole-body control (WBC) policy can stabilize the robot, it cannot distinguish task-relevant interaction forces from forces induced by external disturbances during manipulation. Although force/torque sensors provide direct measurements of physical interactions, retrofitting them entails additional hardware costs and substantial integration effort, particularly for platforms not designed with sensor integration in mind. To address this problem, we propose FWBC-VLA, a force-aware framework that bridges task-level VLA action generation and low-level whole-body compensation control for wheeled-legged robots. First, we introduce HSR-Force, a sensorless residual-torque estimator for inferring contact strength and its temporal variation. These contact estimates are then encoded as tokens and injected into the VLA action expert during action decoding, enabling the policy to perceive contact onset, sustained loading, and release. For loco-manipulation tasks, all parameters of the pretrained VLA backbone are fine-tuned on our WL\&Arm Dataset, which comprises more than 5,000 episodes. Moreover, the robot's proprioceptive state, the Jacobian-derived body-frame force estimate, and the estimated contact state are jointly fed into a compensation generator to produce corrective actions. The manipulation-centric actions are subsequently combined with the corrective actions and passed to the WBC policy for execution. Real-world experiments on whiteboard wiping and door opening with a door closer demonstrate the effectiveness of our FWBC-VLA in contact-rich loco-manipulation.
Tsinghua AI