Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
The paper investigates the inconsistent generalization performance of vision-proprioception policies in robotic manipulation, finding that vision plays a limited role during motion-transition sub-phases due to the policy's preference for proprioceptive signals during training. To address this, they propose Gradient Adjustment with Phase-guidance (GAP), which adaptively modulates the optimization of proprioception based on estimated probabilities of motion-transition phases. Experiments demonstrate that GAP improves the robustness and generalization of vision-proprioception policies in both simulated and real-world environments.
Vision-proprioception policies often fail because proprioception hogs the optimization spotlight during critical motion transitions, but a simple gradient adjustment can restore balance and boost generalization.
Proprioceptive information is critical for precise servo control by providing real-time robotic states. Its collaboration with vision is highly expected to enhance performances of the manipulation policy in complex tasks. However, recent studies have reported inconsistent observations on the generalization of vision-proprioception policies. In this work, we investigate this by conducting temporally controlled experiments. We found that during task sub-phases that robot's motion transitions, which require target localization, the vision modality of the vision-proprioception policy plays a limited role. Further analysis reveals that the policy naturally gravitates toward concise proprioceptive signals that offer faster loss reduction when training, thereby dominating the optimization and suppressing the learning of the visual modality during motion-transition phases. To alleviate this, we propose the Gradient Adjustment with Phase-guidance (GAP) algorithm that adaptively modulates the optimization of proprioception, enabling dynamic collaboration within the vision-proprioception policy. Specifically, we leverage proprioception to capture robotic states and estimate the probability of each timestep in the trajectory belonging to motion-transition phases. During policy learning, we apply fine-grained adjustment that reduces the magnitude of proprioception's gradient based on estimated probabilities, leading to robust and generalizable vision-proprioception policies. The comprehensive experiments demonstrate GAP is applicable in both simulated and real-world environments, across one-arm and dual-arm setups, and compatible with both conventional and Vision-Language-Action models. We believe this work can offer valuable insights into the development of vision-proprioception policies in robotic manipulation.