Search papers, labs, and topics across Lattice.
This paper introduces Foresight, a novel framework for detecting failures in long-horizon robotic manipulation tasks by utilizing action-conditioned world model latents. The approach is significant as it addresses the challenge of ambiguous failure onset and the lack of dense temporal annotations by relying solely on final task-level success or failure labels. Experimental results demonstrate that Foresight outperforms existing failure detection methods across multiple simulation environments and real robotic platforms, highlighting the effectiveness of predictive world-model embeddings for scalable failure monitoring.
Action-conditioned world model embeddings can revolutionize failure detection in long-horizon robotic tasks, achieving reliable monitoring without dense annotations.
Long-horizon tasks are common in real-world robotic deployments, yet failure detection for such tasks remains underexplored. Detecting failures in long-horizon robotic tasks is particularly challenging because failure onset is often ambiguous and dense temporal annotations are typically unavailable. We present Foresight, a failure detection framework that monitors manipulation trajectories using latent representations from an action-conditioned world model. Foresight is trained using only final task-level success or failure labels. By leveraging predictive world-model embeddings, our method provides a unified framework for failure detection across different policies. We further use functional conformal prediction (FCP) to calibrate detection thresholds adaptively. We evaluate Foresight with state-of-the-art vision-language-action policies in simulation on LIBERO-Long, ManiSkill-Long, and BEHAVIOR-1K, compare it against state-of-the-artfailure detection methods, and validate it on real robots with three long-horizon tasks on a ReactorX-200 arm and one task on a Franka arm. Our results suggest that action-conditioned world-model embeddings provide a scalable representation for reliable failure monitoring in long-horizon manipulation.
