Track genomics, neuroscience and molecular biology.
Search papers, labs, and topics across Lattice.
This paper introduces MAIA, a multimodal world model that learns nominal contact-rich manipulation dynamics from vision, action, and force data to predict future visual and force responses. By forecasting these physical interactions, MAIA enables early anticipation of execution anomalies, outperforming reactive methods in both discrimination accuracy and lead time on the new CRIA dataset. Furthermore, integrating MAIA's anomaly predictions into action selection significantly improves end-to-end task success for manipulation policies.
Proactive anomaly anticipation in contact-rich manipulation is achieved by predicting future physical interactions, enabling earlier detection and improved task success.
Reliable contact-rich manipulation requires detecting execution deviations early enough for corrective action. Yet most monitoring methods are reactive, identifying anomalies only after they become observable. Anticipating such anomalies is challenging because visual observations only partially reveal physical interaction, while planned actions describe intended motion rather than their actual physical consequences. We address this problem with MAIA, a multimodal framework for anomaly anticipation. Its vision-action-force world model learns nominal interaction dynamics and predicts the visual and force responses of planned actions, while a lightweight anomaly head predicts impending interaction anomalies. To enable systematic evaluation, we introduce CRIA, a real-world multimodal dataset of 2,852 trajectories spanning 18 contact-rich manipulation tasks, with frame-level annotations of anomaly onset and evolution. Experiments on both seen and novel tasks show that MAIA achieves more accurate anomaly discrimination than representative baselines while providing earlier warnings, with bACC scores of 0.765 and 0.710 and median anticipation lead times of 28.0 and 25.1 steps, respectively. Moreover, using these predictions for anomaly-aware action selection improves end-to-end task success from 52.6% to 61.5% without retraining the underlying manipulation policy. These results highlight the value of modeling future physical consequences for proactive anomaly anticipation.