Search papers, labs, and topics across Lattice.
This paper introduces IMPACT, a novel framework for training interaction-aware world models that addresses the limitations of existing methods by recalibrating the supervision allocation in the denoising objective. By leveraging cross-attention with manipulated-object tokens, IMPACT constructs an interaction map that enhances the training process without relying on external representations or manual annotations. Experimental results demonstrate that IMPACT significantly improves interaction fidelity, physical plausibility, and visual quality in action-conditioned future predictions for embodied agents compared to traditional MSE-trained approaches.
Recalibrating supervision in world model training can drastically enhance interaction fidelity and physical plausibility without the need for external representations.
World models have made remarkable progress in action-conditioned future prediction for embodied agents, yet still struggle to model physically plausible interactions. Existing approaches address this limitation by constraining the generation process with external representations encoding motion, geometry, or semantics. Obtaining these spatiotemporally dense representations typically requires auxiliary estimators or manual annotations, limiting training scalability. We instead revisit the training objective and identify a supervision-allocation mismatch under the globally averaged mean squared error (MSE) denoising objective: prevalent static content dominates the optimization signal, leaving sparse dynamic-object regions critical to interaction generation disproportionately under-supervised. Motivated by this observation, we introduce IMPACT, a scalable Interaction-aware Model training framework with Prior-guided Attention Calibration and Targeting. IMPACT uses cross-attention associated with manipulated-object tokens as an internal spatiotemporal prior for action-conditioned changes. It samples candidate regions from this prior, calibrates them with detached local prediction errors to construct an interaction map, and uses the map to reweight denoising supervision, requiring neither external representations nor inference-time modifications. Extensive experiments on robot-arm and human-hand manipulation, spanning diverse control modalities and DiT backbones, show that IMPACT consistently outperforms the corresponding MSE-trained baselines, improving interaction fidelity, physical plausibility, and visual quality.
Tsinghua AI